[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:07 dist-test-slave-3kk6 krb5kdc[27876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:07 dist-test-slave-3kk6 krb5kdc[27876](info): set up 2 sockets
Aug 07 06:48:07 dist-test-slave-3kk6 krb5kdc[27876](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 06:48:08.712700 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.431s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:08 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:08.937907 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39163
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:09.336108 27892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:09.336603 27892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:09.336771 27892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:09.344758 27892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:09.344946 27892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:09.345064 27892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:09.345184 27892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:09.345268 27892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:09.356576 27892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:39163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:09.358769 27892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:09.360831 27892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:09.381915 27898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:09.381742 27902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:09.381798 27900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:09.386837 27892 server_base.cc:1034] running on GCE node
I20260807 06:48:09.387926 27892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:09.390050 27892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:09.391297 27892 hybrid_clock.cc:648] HybridClock initialized: now 1786085289391216 us; error 100 us; skew 500 ppm
I20260807 06:48:09.391916 27892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:09 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:09.395838 27892 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:09.398391 27892 webserver.cc:460] Webserver started at http://127.27.54.126:45663/ using document root <none> and password file <none>
I20260807 06:48:09.399461 27892 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:09.399648 27892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:09.400081 27892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:09.404237 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b"
format_stamp: "Formatted at 2026-08-07 06:48:09 on dist-test-slave-3kk6"
server_key: "9e5480f45452c0e68fd7189ec82052e0"
server_key_iv: "ca522d5af77adbba3ad5df83f226f033"
server_key_version: "encryptionkey@0"
I20260807 06:48:09.405045 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b"
format_stamp: "Formatted at 2026-08-07 06:48:09 on dist-test-slave-3kk6"
server_key: "9e5480f45452c0e68fd7189ec82052e0"
server_key_iv: "ca522d5af77adbba3ad5df83f226f033"
server_key_version: "encryptionkey@0"
I20260807 06:48:09.413089 27892 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260807 06:48:09.418776 27912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.001s sys 0.000s
I20260807 06:48:09.420976 27892 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260807 06:48:09.421289 27892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b"
format_stamp: "Formatted at 2026-08-07 06:48:09 on dist-test-slave-3kk6"
server_key: "9e5480f45452c0e68fd7189ec82052e0"
server_key_iv: "ca522d5af77adbba3ad5df83f226f033"
server_key_version: "encryptionkey@0"
I20260807 06:48:09.421535 27892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:09.447868 27892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:09.453708 27892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:09.454178 27892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:09.479031 27892 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:39163
I20260807 06:48:09.479080 27986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:39163 every 8 connection(s)
I20260807 06:48:09.481421 27892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:09.488225 27993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:09.492285 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27892
I20260807 06:48:09.492686 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:09.495060 27865 external_mini_cluster.cc:1428] Setting key b47eaade7e78eacca5fd32b4e20a78ca
I20260807 06:48:09.504419 27993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b: Bootstrap starting.
I20260807 06:48:09.510823 27993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:09.512460 27993 log.cc:824] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:09.516880 27993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b: No bootstrap required, opened a new log
I20260807 06:48:09.526042 27993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39163 } }
I20260807 06:48:09.526607 27993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:09.526860 27993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcfbdc1bf584f78b9729d3a5a84db4b, State: Initialized, Role: FOLLOWER
I20260807 06:48:09.528137 27993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39163 } }
I20260807 06:48:09.528477 27993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:09.528785 27993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:09.529004 27993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 0 FOLLOWER]: Advancing to term 1
Aug 07 06:48:09 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:09.531666 27993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39163 } }
I20260807 06:48:09.532712 27993 leader_election.cc:302] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcfbdc1bf584f78b9729d3a5a84db4b; no voters:
I20260807 06:48:09.533691 27993 leader_election.cc:288] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:09.534040 27999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:09.537111 27993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:09.541157 27999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [term 1 LEADER]: Becoming Leader. State: Replica: 3fcfbdc1bf584f78b9729d3a5a84db4b, State: Running, Role: LEADER
I20260807 06:48:09.542506 27999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39163 } }
I20260807 06:48:09.550242 27995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:09.501107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.27.54.126:39163)
0807 06:48:09.502381 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:09.502394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:09.506498 (+ 4104us) server_negotiation.cc:407] Connection header received
0807 06:48:09.516567 (+ 10069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:09.516589 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:09.517318 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:09.517785 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:09.521302 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:09.522798 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:09.524528 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:09.525281 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:09.525571 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:09.532366 (+ 6795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:09.532426 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:09.532457 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:09.532507 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:09.538760 (+ 6253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:09.540512 (+ 1752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:09.540525 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:09.540538 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:09.540904 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:09.541816 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:09.541823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:09.541830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:09.543180 (+ 1350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:09.543730 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:09.547597 (+ 3867us) server_negotiation.cc:299] Negotiation successful
0807 06:48:09.548174 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":551,"threads_started":1}
I20260807 06:48:09.550150 28000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39163 } } }
I20260807 06:48:09.550879 28000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:09.550184 28001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fcfbdc1bf584f78b9729d3a5a84db4b. Latest consensus state: current_term: 1 leader_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcfbdc1bf584f78b9729d3a5a84db4b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39163 } } }
I20260807 06:48:09.551892 28001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:09.558584 28008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:09.566150 28008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:09.580539 28008 catalog_manager.cc:1347] Generated new cluster ID: 3ec41947fa044c7895937a7b856ca4e7
I20260807 06:48:09.580693 28008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:09.602049 28008 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:09.603645 28008 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:09.610884 28008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fcfbdc1bf584f78b9729d3a5a84db4b: Generated new TSK 0
I20260807 06:48:09.611956 28008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:09.690471 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39163
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:10.143498 28021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:10.144081 28021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:10.144481 28021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:10.156066 28021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:10.156378 28021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:10.156643 28021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:10.173982 28021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:10.176178 28021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:10.179677 28021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:10.203681 28030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:10.203876 28034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:10.203652 28031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:10.207953 28021 server_base.cc:1034] running on GCE node
I20260807 06:48:10.209630 28021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:10.211692 28021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:10.215268 28021 hybrid_clock.cc:648] HybridClock initialized: now 1786085290215171 us; error 99 us; skew 500 ppm
I20260807 06:48:10.216137 28021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:10 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:10.225544 28021 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:10.231433 28021 webserver.cc:460] Webserver started at http://127.27.54.65:43921/ using document root <none> and password file <none>
I20260807 06:48:10.234077 28021 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:10.234469 28021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:10.235481 28021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:10.243834 28021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "733c54da68644041aa75ebdcd7f90058"
format_stamp: "Formatted at 2026-08-07 06:48:10 on dist-test-slave-3kk6"
server_key: "c373b91d6c9d7e797ccdbf91b3818692"
server_key_iv: "40f0d6024a89020c971ad36775ea1f4c"
server_key_version: "encryptionkey@0"
I20260807 06:48:10.247541 28021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "733c54da68644041aa75ebdcd7f90058"
format_stamp: "Formatted at 2026-08-07 06:48:10 on dist-test-slave-3kk6"
server_key: "c373b91d6c9d7e797ccdbf91b3818692"
server_key_iv: "40f0d6024a89020c971ad36775ea1f4c"
server_key_version: "encryptionkey@0"
I20260807 06:48:10.266870 28021 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.018s sys 0.000s
I20260807 06:48:10.278102 28043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:10.283658 28021 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.003s
I20260807 06:48:10.284333 28021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "733c54da68644041aa75ebdcd7f90058"
format_stamp: "Formatted at 2026-08-07 06:48:10 on dist-test-slave-3kk6"
server_key: "c373b91d6c9d7e797ccdbf91b3818692"
server_key_iv: "40f0d6024a89020c971ad36775ea1f4c"
server_key_version: "encryptionkey@0"
I20260807 06:48:10.285470 28021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:10.330914 28021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:10.349267 28021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:10.350939 28021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:10.355098 28021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:10.361934 28021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:10.362435 28021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:10.363113 28021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:10.363410 28021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:10.491055 28021 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43617
I20260807 06:48:10.491206 28203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43617 every 8 connection(s)
I20260807 06:48:10.500885 28021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:10.503682 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28021
I20260807 06:48:10.504478 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:10.510452 27865 external_mini_cluster.cc:1428] Setting key e959933746b7545356e795bb99abacb8
Aug 07 06:48:10 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
I20260807 06:48:10.592784 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:10.520537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:47749 (local address 127.27.54.126:39163)
0807 06:48:10.522560 (+ 2023us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:10.522600 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:10.528670 (+ 6070us) server_negotiation.cc:407] Connection header received
0807 06:48:10.545428 (+ 16758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:10.545462 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:10.546037 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:10.546582 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:10.552916 (+ 6334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:10.555069 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:10.559151 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:10.560492 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:10.561009 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:10.573810 (+ 12801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:10.573850 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:10.573878 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:10.573984 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:10.582430 (+ 8446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:10.584156 (+ 1726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:10.584176 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:10.584191 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:10.584659 (+ 468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:10.586245 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:10.586251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:10.586261 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:10.587369 (+ 1108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:10.588190 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:10.591003 (+ 2813us) server_negotiation.cc:299] Negotiation successful
0807 06:48:10.591850 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":961,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
I20260807 06:48:10.599200 28210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:10.523314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39163 (local address 127.27.54.65:47749)
0807 06:48:10.527622 (+ 4308us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:10.528057 (+ 435us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:10.542937 (+ 14880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:10.549669 (+ 6732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:10.549717 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:10.550895 (+ 1178us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:10.552041 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:10.552115 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:10.555601 (+ 3486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:10.555614 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:10.558948 (+ 3334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:10.558966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:10.561590 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:10.561634 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:10.562813 (+ 1179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:10.565086 (+ 2273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:10.565200 (+ 114us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:10.573079 (+ 7879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:10.582932 (+ 9853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:10.582950 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:10.583001 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:10.583913 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:10.585065 (+ 1152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:10.585077 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:10.585080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:10.585780 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:10.587974 (+ 2194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:10.588049 (+ 75us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:10.589234 (+ 1185us) client_negotiation.cc:769] Sending connection context
0807 06:48:10.590395 (+ 1161us) client_negotiation.cc:240] Negotiation successful
0807 06:48:10.592067 (+ 1672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2528,"thread_start_us":1689,"threads_started":1}
I20260807 06:48:10.606447 28205 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39163
I20260807 06:48:10.608239 28205 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:10.612365 28205 heartbeater.cc:511] Master 127.27.54.126:39163 requested a full tablet report, sending...
I20260807 06:48:10.628938 27934 ts_manager.cc:194] Registered new tserver with Master: 733c54da68644041aa75ebdcd7f90058 (127.27.54.65:43617)
I20260807 06:48:10.643304 27934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:47749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
W20260807 06:48:10.692431 27865 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.54.66: real 0.166s user 0.003s sys 0.044s
I20260807 06:48:10.695904 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39163
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:11.156728 28215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:11.157097 28215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:11.157210 28215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:11.164872 28215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:11.165015 28215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:11.165145 28215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:11.175374 28215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:11.176990 28215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:11.178464 28215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:11.193454 28222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:11.193373 28223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:11.193481 28225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:11.194499 28215 server_base.cc:1034] running on GCE node
I20260807 06:48:11.195295 28215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:11.196439 28215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:11.197762 28215 hybrid_clock.cc:648] HybridClock initialized: now 1786085291197676 us; error 56 us; skew 500 ppm
I20260807 06:48:11.198154 28215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:11.201786 28215 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:11.203650 28215 webserver.cc:460] Webserver started at http://127.27.54.66:37671/ using document root <none> and password file <none>
I20260807 06:48:11.204384 28215 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:11.204499 28215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:11.204790 28215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:11.208074 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "a620fcda757a456f8ad91ce3723dde9a"
format_stamp: "Formatted at 2026-08-07 06:48:11 on dist-test-slave-3kk6"
server_key: "2dff8dc376258987968b455c46d3a98d"
server_key_iv: "9f8a3d4434ad9668ca7f5ff2badf814d"
server_key_version: "encryptionkey@0"
I20260807 06:48:11.208890 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "a620fcda757a456f8ad91ce3723dde9a"
format_stamp: "Formatted at 2026-08-07 06:48:11 on dist-test-slave-3kk6"
server_key: "2dff8dc376258987968b455c46d3a98d"
server_key_iv: "9f8a3d4434ad9668ca7f5ff2badf814d"
server_key_version: "encryptionkey@0"
I20260807 06:48:11.214789 28215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:48:11.218636 28234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:11.220283 28215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:11.220479 28215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "a620fcda757a456f8ad91ce3723dde9a"
format_stamp: "Formatted at 2026-08-07 06:48:11 on dist-test-slave-3kk6"
server_key: "2dff8dc376258987968b455c46d3a98d"
server_key_iv: "9f8a3d4434ad9668ca7f5ff2badf814d"
server_key_version: "encryptionkey@0"
I20260807 06:48:11.220722 28215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:11.238579 28215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:11.243292 28215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:11.243625 28215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:11.244700 28215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:11.246203 28215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:11.246284 28215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:11.246433 28215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:11.246490 28215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:11.283993 28215 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:37643
I20260807 06:48:11.284055 28394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:37643 every 8 connection(s)
I20260807 06:48:11.285375 28215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:11.291337 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28215
I20260807 06:48:11.291538 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:11.292184 27865 external_mini_cluster.cc:1428] Setting key 07d5a7e95c0fa3adbca16f766cf983a7
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:11.312228 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.288278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:33539 (local address 127.27.54.126:39163)
0807 06:48:11.288835 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:11.288840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:11.289579 (+ 739us) server_negotiation.cc:407] Connection header received
0807 06:48:11.291976 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:11.291981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:11.292052 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:11.292156 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:11.293922 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.294837 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.296460 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.296986 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.297133 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.305678 (+ 8545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:11.305684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:11.305688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:11.305726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:11.308742 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.309598 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.309610 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.309613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.309762 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.310257 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.310263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.310266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.310566 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:11.310781 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:11.311550 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 06:48:11.311819 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
I20260807 06:48:11.313498 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.288624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39163 (local address 127.27.54.66:33539)
0807 06:48:11.289389 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:11.289427 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:11.291643 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:11.292469 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:11.292480 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:11.293151 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:11.293732 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:11.293750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.295022 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.295027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:11.296296 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:11.296310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.297248 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.297256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:11.297500 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.302829 (+ 5329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:11.302856 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:11.305277 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:11.308941 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:11.308955 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:11.308970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:11.309449 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:11.309893 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:11.309898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:11.309902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:11.310121 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:11.310748 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:11.310757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:11.311155 (+ 398us) client_negotiation.cc:769] Sending connection context
0807 06:48:11.311448 (+ 293us) client_negotiation.cc:240] Negotiation successful
0807 06:48:11.312061 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
I20260807 06:48:11.315336 28395 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39163
I20260807 06:48:11.315673 28395 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:11.316519 28395 heartbeater.cc:511] Master 127.27.54.126:39163 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
I20260807 06:48:11.318260 27934 ts_manager.cc:194] Registered new tserver with Master: a620fcda757a456f8ad91ce3723dde9a (127.27.54.66:37643)
I20260807 06:48:11.319204 27934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:33539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:11.346268 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39163
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:48:11.668219 28205 heartbeater.cc:503] Master 127.27.54.126:39163 was elected leader, sending a full tablet report...
W20260807 06:48:11.705431 28402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:11.705755 28402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:11.705860 28402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:11.713313 28402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:11.713443 28402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:11.713567 28402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:11.723999 28402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:11.725459 28402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:11.727000 28402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:11.741077 28408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:11.741197 28409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:11.741219 28412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:11.742424 28402 server_base.cc:1034] running on GCE node
I20260807 06:48:11.743312 28402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:11.744695 28402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:11.745945 28402 hybrid_clock.cc:648] HybridClock initialized: now 1786085291745882 us; error 54 us; skew 500 ppm
I20260807 06:48:11.746301 28402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:11.749676 28402 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:11.751561 28402 webserver.cc:460] Webserver started at http://127.27.54.67:38525/ using document root <none> and password file <none>
I20260807 06:48:11.752332 28402 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:11.752496 28402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:11.752863 28402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:11.756669 28402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "012c50dd63a6457fa23bd9167a33e91c"
format_stamp: "Formatted at 2026-08-07 06:48:11 on dist-test-slave-3kk6"
server_key: "99c1a98a0183990c4036c33e350e00c9"
server_key_iv: "4c6b86fa417f90597236faba9b8a494b"
server_key_version: "encryptionkey@0"
I20260807 06:48:11.757634 28402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "012c50dd63a6457fa23bd9167a33e91c"
format_stamp: "Formatted at 2026-08-07 06:48:11 on dist-test-slave-3kk6"
server_key: "99c1a98a0183990c4036c33e350e00c9"
server_key_iv: "4c6b86fa417f90597236faba9b8a494b"
server_key_version: "encryptionkey@0"
I20260807 06:48:11.763687 28402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260807 06:48:11.767500 28423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:11.768898 28402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:11.769099 28402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "012c50dd63a6457fa23bd9167a33e91c"
format_stamp: "Formatted at 2026-08-07 06:48:11 on dist-test-slave-3kk6"
server_key: "99c1a98a0183990c4036c33e350e00c9"
server_key_iv: "4c6b86fa417f90597236faba9b8a494b"
server_key_version: "encryptionkey@0"
I20260807 06:48:11.769335 28402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:11.788208 28402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:11.792939 28402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:11.793274 28402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:11.794377 28402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:11.796000 28402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:11.796110 28402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:11.796275 28402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:11.796351 28402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:11.837703 28402 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:35329
I20260807 06:48:11.837755 28575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:35329 every 8 connection(s)
I20260807 06:48:11.839308 28402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:11.842875 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28402
I20260807 06:48:11.843060 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:11.843549 27865 external_mini_cluster.cc:1428] Setting key b3eb83a02ba9b3266a1ce9141f242ae3
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:11.861817 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.842123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:49851 (local address 127.27.54.126:39163)
0807 06:48:11.842760 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:11.842768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:11.844062 (+ 1294us) server_negotiation.cc:407] Connection header received
0807 06:48:11.846209 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:11.846218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:11.846336 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:11.846469 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:11.848150 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.849243 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.850836 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.851318 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.851460 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.855107 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:11.855113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:11.855116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:11.855153 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:11.858148 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.859015 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.859024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.859029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.859223 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.859779 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.859784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.859787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.860222 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:11.860396 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:11.861167 (+ 771us) server_negotiation.cc:299] Negotiation successful
0807 06:48:11.861399 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":270,"threads_started":1}
I20260807 06:48:11.862860 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.842796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39163 (local address 127.27.54.67:49851)
0807 06:48:11.843805 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:11.843861 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:11.845890 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:11.846795 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:11.846804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:11.847346 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:11.847949 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:11.847975 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.849400 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.849404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:11.850636 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:11.850707 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.851583 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.851594 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:11.851832 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.852514 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:11.852541 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:11.854877 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:11.858284 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:11.858291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:11.858303 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:11.858807 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:11.859356 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:11.859359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:11.859361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:11.859651 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:11.860361 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:11.860367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:11.860790 (+ 423us) client_negotiation.cc:769] Sending connection context
0807 06:48:11.861067 (+ 277us) client_negotiation.cc:240] Negotiation successful
0807 06:48:11.861620 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":523,"threads_started":1}
I20260807 06:48:11.864288 28576 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39163
I20260807 06:48:11.864555 28576 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:11.865322 28576 heartbeater.cc:511] Master 127.27.54.126:39163 requested a full tablet report, sending...
I20260807 06:48:11.866818 27934 ts_manager.cc:194] Registered new tserver with Master: 012c50dd63a6457fa23bd9167a33e91c (127.27.54.67:35329)
I20260807 06:48:11.867718 27934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:49851
I20260807 06:48:11.873948 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:11.933764 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.916694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.27.54.65:43617)
0807 06:48:11.917455 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:11.917464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:11.917501 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:48:11.917591 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:11.917598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:11.917858 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:11.918066 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:11.919708 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.920763 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.922111 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.922794 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.922938 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.927470 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:11.927483 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:11.927499 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:11.927535 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:11.930571 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.931331 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.931347 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.931349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.931541 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.931969 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.931974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.931977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.932409 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:11.932765 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:11.933105 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 06:48:11.933384 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":380,"threads_started":1}
W20260807 06:48:11.935421 28086 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60458
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:48:11 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:11.989655 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.974361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.27.54.65:43617)
0807 06:48:11.974702 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:11.974713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:11.975006 (+ 293us) server_negotiation.cc:407] Connection header received
0807 06:48:11.975253 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:11.975259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:11.975335 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:11.975448 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:11.976962 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.977683 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.979192 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.979648 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.979792 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.984181 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:11.984187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:11.984190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:11.984229 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:11.986943 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.987637 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.987644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.987652 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.987800 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:11.988315 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:11.988320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:11.988322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:11.988591 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:11.988783 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:11.989078 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 06:48:11.989270 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:48:11.991811 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28021
I20260807 06:48:12.004850 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:11.992897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.27.54.65:43617)
0807 06:48:11.993134 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:11.993140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:11.993398 (+ 258us) server_negotiation.cc:407] Connection header received
0807 06:48:11.993514 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:11.993518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:11.993574 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:11.993652 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:11.994893 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.995612 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.996935 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:11.997377 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:11.997471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:11.999899 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:11.999903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:11.999907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:11.999936 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:12.002476 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:12.003145 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:12.003148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:12.003151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:12.003288 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:12.003618 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:12.003621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:12.003623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:12.003873 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:12.004053 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:12.004296 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 06:48:12.004463 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:48:12.005642 28199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:12.095305 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28021
I20260807 06:48:12.122212 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28215
Aug 07 06:48:12 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:12.139550 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:12.123259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.27.54.66:37643)
0807 06:48:12.123853 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:12.123860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:12.123888 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:48:12.123955 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:12.123963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:12.124178 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:12.124327 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:12.125715 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:12.127110 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:12.128416 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:12.128955 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:12.129063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:12.133471 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:12.133485 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:12.133499 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:12.133535 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:12.136642 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:12.137363 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:12.137370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:12.137373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:12.137498 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:12.137906 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:12.137913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:12.137916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:12.138338 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:12.138610 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:12.139022 (+ 412us) server_negotiation.cc:299] Negotiation successful
0807 06:48:12.139230 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":306,"threads_started":1}
I20260807 06:48:12.140839 28389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:12.225054 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28215
I20260807 06:48:12.248895 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28402
Aug 07 06:48:12 dist-test-slave-3kk6 krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:12.269021 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:12.250027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.27.54.67:35329)
0807 06:48:12.250848 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:12.250855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:12.250888 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:48:12.250964 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:12.250972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:12.251190 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:12.251407 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:12.253096 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:12.254292 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:12.255990 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:12.256738 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:12.256903 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:12.261886 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:12.261899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:12.261916 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:12.261963 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:12.265301 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:12.266160 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:12.266169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:12.266172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:12.266341 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:12.267057 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:12.267064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:12.267070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:12.267499 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:12.267886 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:12.268281 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 06:48:12.268621 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":405,"threads_started":1}
I20260807 06:48:12.270470 28570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:12.355042 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28402
I20260807 06:48:12.384070 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27892
I20260807 06:48:12.385356 27981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:12.492146 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27892
2026-08-07T06:48:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5277 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:12 dist-test-slave-3kk6 krb5kdc[28606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:12 dist-test-slave-3kk6 krb5kdc[28606](info): set up 2 sockets
Aug 07 06:48:12 dist-test-slave-3kk6 krb5kdc[28606](info): commencing operation
krb5kdc: starting...
W20260807 06:48:13.805617 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:13 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:14.031589 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:32771
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:32771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:14.379963 28623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:14.380270 28623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:14.380434 28623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:14.387868 28623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:14.387987 28623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:14.388098 28623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:14.388204 28623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:14.388275 28623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:14.398435 28623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:32771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:14.399868 28623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:14.401222 28623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:14.413816 28632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:14.413791 28635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:14.413882 28633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:14.414606 28623 server_base.cc:1034] running on GCE node
I20260807 06:48:14.415509 28623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:14.416977 28623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:14.418218 28623 hybrid_clock.cc:648] HybridClock initialized: now 1786085294418173 us; error 61 us; skew 500 ppm
I20260807 06:48:14.418567 28623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:14 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:14.421912 28623 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:14.423467 28623 webserver.cc:460] Webserver started at http://127.27.54.126:34949/ using document root <none> and password file <none>
I20260807 06:48:14.424126 28623 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:14.424230 28623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:14.424499 28623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:14.427595 28623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "6616632624244f1e9d7e110b85e7aa7b"
format_stamp: "Formatted at 2026-08-07 06:48:14 on dist-test-slave-3kk6"
server_key: "89b62f3fa40055bd6d52eb5b7587c53f"
server_key_iv: "93b9628cbae51f76f775d060064fbee5"
server_key_version: "encryptionkey@0"
I20260807 06:48:14.428318 28623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "6616632624244f1e9d7e110b85e7aa7b"
format_stamp: "Formatted at 2026-08-07 06:48:14 on dist-test-slave-3kk6"
server_key: "89b62f3fa40055bd6d52eb5b7587c53f"
server_key_iv: "93b9628cbae51f76f775d060064fbee5"
server_key_version: "encryptionkey@0"
I20260807 06:48:14.433506 28623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:48:14.437052 28644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:14.438377 28623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 06:48:14.438567 28623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "6616632624244f1e9d7e110b85e7aa7b"
format_stamp: "Formatted at 2026-08-07 06:48:14 on dist-test-slave-3kk6"
server_key: "89b62f3fa40055bd6d52eb5b7587c53f"
server_key_iv: "93b9628cbae51f76f775d060064fbee5"
server_key_version: "encryptionkey@0"
I20260807 06:48:14.438830 28623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:14.461934 28623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:14.466552 28623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:14.466938 28623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:14.485806 28623 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:32771
I20260807 06:48:14.485837 28704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:32771 every 8 connection(s)
I20260807 06:48:14.487408 28623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:14.491974 28705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:14.495340 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28623
I20260807 06:48:14.495570 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:14.496253 27865 external_mini_cluster.cc:1428] Setting key a39c05158e2a7f974778c1715fadef15
I20260807 06:48:14.505887 28705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b: Bootstrap starting.
I20260807 06:48:14.510488 28705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b: Neither blocks nor log segments found. Creating new log.
Aug 07 06:48:14 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:14.511956 28705 log.cc:824] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:14.516008 28705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b: No bootstrap required, opened a new log
I20260807 06:48:14.522745 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:14.498493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.27.54.126:32771)
0807 06:48:14.499430 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:14.499442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:14.499485 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:48:14.501233 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:14.501257 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:14.501688 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:14.502040 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:14.504087 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:14.505416 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:14.507232 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:14.507942 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:14.508091 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:14.513382 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:14.513390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:14.513405 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:14.513470 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:14.517577 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:14.518413 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:14.518423 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:14.518432 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:14.518684 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:14.519122 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:14.519127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:14.519131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:14.519948 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:14.520269 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:14.520755 (+ 486us) server_negotiation.cc:299] Negotiation successful
0807 06:48:14.521236 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":395,"threads_started":1}
I20260807 06:48:14.526937 28705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6616632624244f1e9d7e110b85e7aa7b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 32771 } }
I20260807 06:48:14.527323 28705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:14.527410 28705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6616632624244f1e9d7e110b85e7aa7b, State: Initialized, Role: FOLLOWER
I20260807 06:48:14.527974 28705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6616632624244f1e9d7e110b85e7aa7b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 32771 } }
I20260807 06:48:14.528211 28705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:14.528328 28705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:14.528455 28705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:14.530733 28705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6616632624244f1e9d7e110b85e7aa7b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 32771 } }
I20260807 06:48:14.531325 28705 leader_election.cc:302] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6616632624244f1e9d7e110b85e7aa7b; no voters:
I20260807 06:48:14.532101 28705 leader_election.cc:288] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:14.532232 28712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:14.533128 28712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [term 1 LEADER]: Becoming Leader. State: Replica: 6616632624244f1e9d7e110b85e7aa7b, State: Running, Role: LEADER
I20260807 06:48:14.533674 28712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6616632624244f1e9d7e110b85e7aa7b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 32771 } }
I20260807 06:48:14.534421 28705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:14.536870 28714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6616632624244f1e9d7e110b85e7aa7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6616632624244f1e9d7e110b85e7aa7b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 32771 } } }
I20260807 06:48:14.537268 28714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:14.537212 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6616632624244f1e9d7e110b85e7aa7b. Latest consensus state: current_term: 1 leader_uuid: "6616632624244f1e9d7e110b85e7aa7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6616632624244f1e9d7e110b85e7aa7b" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 32771 } } }
I20260807 06:48:14.537528 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:14.538287 28721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:14.546008 28721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:14.556268 28721 catalog_manager.cc:1347] Generated new cluster ID: 11eee2b378f049a882668b4cb6a2a3b1
I20260807 06:48:14.556407 28721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:14.567512 28721 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:14.568866 28721 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:14.579588 28721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6616632624244f1e9d7e110b85e7aa7b: Generated new TSK 0
I20260807 06:48:14.580623 28721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:14.646876 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:32771
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:15.001816 28746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:15.002159 28746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:15.002267 28746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:15.009935 28746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:15.010064 28746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:15.010187 28746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:15.020503 28746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:32771
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:15.022073 28746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:15.023502 28746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:15.037534 28752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:15.037585 28753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:15.037608 28756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:15.038477 28746 server_base.cc:1034] running on GCE node
I20260807 06:48:15.039218 28746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:15.040289 28746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:15.041538 28746 hybrid_clock.cc:648] HybridClock initialized: now 1786085295041493 us; error 40 us; skew 500 ppm
I20260807 06:48:15.041882 28746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:15 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:15.045234 28746 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:15.046849 28746 webserver.cc:460] Webserver started at http://127.27.54.65:44045/ using document root <none> and password file <none>
I20260807 06:48:15.047554 28746 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:15.047716 28746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:15.048043 28746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:15.051151 28746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "bdfd5868f2704d2985274259b006b58d"
format_stamp: "Formatted at 2026-08-07 06:48:15 on dist-test-slave-3kk6"
server_key: "11e42c1b1f4a5aacd969316aa4f96eaa"
server_key_iv: "ba7a6c503a7148b0bd4ab508dc3420f0"
server_key_version: "encryptionkey@0"
I20260807 06:48:15.052029 28746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "bdfd5868f2704d2985274259b006b58d"
format_stamp: "Formatted at 2026-08-07 06:48:15 on dist-test-slave-3kk6"
server_key: "11e42c1b1f4a5aacd969316aa4f96eaa"
server_key_iv: "ba7a6c503a7148b0bd4ab508dc3420f0"
server_key_version: "encryptionkey@0"
I20260807 06:48:15.057379 28746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 06:48:15.061192 28763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:15.062614 28746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:48:15.062901 28746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "bdfd5868f2704d2985274259b006b58d"
format_stamp: "Formatted at 2026-08-07 06:48:15 on dist-test-slave-3kk6"
server_key: "11e42c1b1f4a5aacd969316aa4f96eaa"
server_key_iv: "ba7a6c503a7148b0bd4ab508dc3420f0"
server_key_version: "encryptionkey@0"
I20260807 06:48:15.063127 28746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:15.085592 28746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:15.090139 28746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:15.090458 28746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:15.091583 28746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:15.093063 28746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:15.093173 28746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:15.093324 28746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:15.093397 28746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:15.128036 28746 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:36233
I20260807 06:48:15.128089 28911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:36233 every 8 connection(s)
I20260807 06:48:15.129390 28746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:15.130824 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28746
I20260807 06:48:15.131034 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:15.131843 27865 external_mini_cluster.cc:1428] Setting key 3bce063135607086f3431b408ed34480
Aug 07 06:48:15 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:15.152377 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:15.132442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:53553 (local address 127.27.54.126:32771)
0807 06:48:15.132921 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:15.132926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:15.134126 (+ 1200us) server_negotiation.cc:407] Connection header received
0807 06:48:15.136296 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:15.136305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:15.136397 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:15.136516 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:15.138018 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.139234 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.140940 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.141585 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.141733 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:15.146052 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:15.146057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:15.146061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:15.146095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:15.148831 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:15.149827 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:15.149831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:15.149833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:15.149959 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:15.150495 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:15.150499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:15.150501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:15.150828 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:15.150946 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:15.151939 (+ 993us) server_negotiation.cc:299] Negotiation successful
0807 06:48:15.152112 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
I20260807 06:48:15.154068 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:15.133091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:32771 (local address 127.27.54.65:53553)
0807 06:48:15.133947 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:15.134002 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:15.136014 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:15.136727 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:15.136736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:15.137274 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:15.137835 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:15.137852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.139384 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.139388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:15.140799 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:15.140812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.141873 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.141882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:15.142176 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:15.143014 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:15.143038 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:15.145808 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:15.148984 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:15.148995 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:15.149014 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:15.149701 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:15.150078 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:15.150087 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:15.150092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:15.150381 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:15.150962 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:15.150974 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:15.151462 (+ 488us) client_negotiation.cc:769] Sending connection context
0807 06:48:15.151857 (+ 395us) client_negotiation.cc:240] Negotiation successful
0807 06:48:15.152462 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":364,"threads_started":1}
I20260807 06:48:15.155807 28913 heartbeater.cc:347] Connected to a master server at 127.27.54.126:32771
I20260807 06:48:15.156141 28913 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
I20260807 06:48:15.156961 28913 heartbeater.cc:511] Master 127.27.54.126:32771 requested a full tablet report, sending...
I20260807 06:48:15.159453 28662 ts_manager.cc:194] Registered new tserver with Master: bdfd5868f2704d2985274259b006b58d (127.27.54.65:36233)
I20260807 06:48:15.161788 28662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:53553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:15.184285 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:32771
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:15.533142 28922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:15.533517 28922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:15.533697 28922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:15.541704 28922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:15.541896 28922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:15.542080 28922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:15.552826 28922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:32771
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:15.554342 28922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:15.555809 28922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:15.570920 28928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:15.570933 28932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:15.571147 28927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:15.571730 28922 server_base.cc:1034] running on GCE node
I20260807 06:48:15.572453 28922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:15.573501 28922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:15.574796 28922 hybrid_clock.cc:648] HybridClock initialized: now 1786085295574724 us; error 48 us; skew 500 ppm
I20260807 06:48:15.575196 28922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:15 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:15.578909 28922 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:15.581004 28922 webserver.cc:460] Webserver started at http://127.27.54.66:45437/ using document root <none> and password file <none>
I20260807 06:48:15.581871 28922 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:15.582047 28922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:15.582448 28922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:15.586189 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "0bd01238ed8b42889e889d630b548e87"
format_stamp: "Formatted at 2026-08-07 06:48:15 on dist-test-slave-3kk6"
server_key: "f607e72cb4022dff0ea878cf4472f818"
server_key_iv: "b0135c0501f7219cde09d458f395b3cf"
server_key_version: "encryptionkey@0"
I20260807 06:48:15.587272 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "0bd01238ed8b42889e889d630b548e87"
format_stamp: "Formatted at 2026-08-07 06:48:15 on dist-test-slave-3kk6"
server_key: "f607e72cb4022dff0ea878cf4472f818"
server_key_iv: "b0135c0501f7219cde09d458f395b3cf"
server_key_version: "encryptionkey@0"
I20260807 06:48:15.593389 28922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 06:48:15.597141 28943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:15.598589 28922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:48:15.598834 28922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "0bd01238ed8b42889e889d630b548e87"
format_stamp: "Formatted at 2026-08-07 06:48:15 on dist-test-slave-3kk6"
server_key: "f607e72cb4022dff0ea878cf4472f818"
server_key_iv: "b0135c0501f7219cde09d458f395b3cf"
server_key_version: "encryptionkey@0"
I20260807 06:48:15.599073 28922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:15.627514 28922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:15.632683 28922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:15.633045 28922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:15.634232 28922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:15.635941 28922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:15.636057 28922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:15.636231 28922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:15.636322 28922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:15.673753 28922 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:36311
I20260807 06:48:15.673848 29103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:36311 every 8 connection(s)
I20260807 06:48:15.675307 28922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:15.678364 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28922
I20260807 06:48:15.678563 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:15.679363 27865 external_mini_cluster.cc:1428] Setting key dc2dcd069e2807d5248252e56e58d232
Aug 07 06:48:15 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:15.699177 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:15.678525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:60827 (local address 127.27.54.126:32771)
0807 06:48:15.683112 (+ 4587us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:15.683118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:15.683135 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:48:15.683186 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:15.683190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:15.683297 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:15.683407 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:15.685302 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.686555 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.688306 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.689076 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.689229 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:15.693067 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:15.693072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:15.693076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:15.693109 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:15.695843 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:15.696666 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:15.696670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:15.696673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:15.696814 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:15.697297 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:15.697301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:15.697303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:15.697583 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:15.697745 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:15.698489 (+ 744us) server_negotiation.cc:299] Negotiation successful
0807 06:48:15.698775 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4453,"thread_start_us":4324,"threads_started":1}
I20260807 06:48:15.700230 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:15.679120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:32771 (local address 127.27.54.66:60827)
0807 06:48:15.680093 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:15.680271 (+ 178us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:15.682114 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:15.683779 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:15.683791 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:15.684480 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:15.685090 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:15.685108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.686820 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.686824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:15.688132 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:15.688146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:15.689331 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:15.689337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:15.689551 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:15.690319 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:15.690344 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:15.692813 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:15.696010 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:15.696020 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:15.696044 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:15.696505 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:15.696941 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:15.696946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:15.696949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:15.697154 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:15.697742 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:15.697752 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:15.698109 (+ 357us) client_negotiation.cc:769] Sending connection context
0807 06:48:15.698387 (+ 278us) client_negotiation.cc:240] Negotiation successful
0807 06:48:15.699006 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":475,"threads_started":1}
I20260807 06:48:15.701974 29104 heartbeater.cc:347] Connected to a master server at 127.27.54.126:32771
I20260807 06:48:15.702323 29104 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:15.703261 29104 heartbeater.cc:511] Master 127.27.54.126:32771 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
I20260807 06:48:15.704944 28662 ts_manager.cc:194] Registered new tserver with Master: 0bd01238ed8b42889e889d630b548e87 (127.27.54.66:36311)
I20260807 06:48:15.705966 28662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:60827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:15.734750 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:32771
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:16.084342 29115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:16.084704 29115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:16.084810 29115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:16.092586 29115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:16.092730 29115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:16.092868 29115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:16.107584 29115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:32771
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:16.108965 29115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:16.110369 29115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:16.124730 29121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:16.124814 29125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:16.124766 29122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:16.125609 29115 server_base.cc:1034] running on GCE node
I20260807 06:48:16.126312 29115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:16.127533 29115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:16.128840 29115 hybrid_clock.cc:648] HybridClock initialized: now 1786085296128782 us; error 66 us; skew 500 ppm
I20260807 06:48:16.129189 29115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:16.136124 29115 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:16.137890 29115 webserver.cc:460] Webserver started at http://127.27.54.67:44591/ using document root <none> and password file <none>
I20260807 06:48:16.138718 29115 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:16.138893 29115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:16.139245 29115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:16.142572 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b"
format_stamp: "Formatted at 2026-08-07 06:48:16 on dist-test-slave-3kk6"
server_key: "f172d1f0608eea3e406233e6ff9c9278"
server_key_iv: "3be6d35ca35e5f256a646d11b9844b46"
server_key_version: "encryptionkey@0"
I20260807 06:48:16.143585 29115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b"
format_stamp: "Formatted at 2026-08-07 06:48:16 on dist-test-slave-3kk6"
server_key: "f172d1f0608eea3e406233e6ff9c9278"
server_key_iv: "3be6d35ca35e5f256a646d11b9844b46"
server_key_version: "encryptionkey@0"
I20260807 06:48:16.149121 29115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:48:16.152878 29133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:16.154392 29115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:48:16.154605 29115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b"
format_stamp: "Formatted at 2026-08-07 06:48:16 on dist-test-slave-3kk6"
server_key: "f172d1f0608eea3e406233e6ff9c9278"
server_key_iv: "3be6d35ca35e5f256a646d11b9844b46"
server_key_version: "encryptionkey@0"
I20260807 06:48:16.154968 29115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:16.165125 28913 heartbeater.cc:503] Master 127.27.54.126:32771 was elected leader, sending a full tablet report...
I20260807 06:48:16.180500 29115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:16.185360 29115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:16.185694 29115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:16.186862 29115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:16.188330 29115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:16.188441 29115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:16.188591 29115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:16.188664 29115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:16.226055 29115 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:40717
I20260807 06:48:16.226099 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:40717 every 8 connection(s)
I20260807 06:48:16.227555 29115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:16.229346 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 29115
I20260807 06:48:16.229543 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:16.230046 27865 external_mini_cluster.cc:1428] Setting key db58fbda4aa4c0146a4819ccd5b6b852
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:16.251087 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.230763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:38453 (local address 127.27.54.126:32771)
0807 06:48:16.231285 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.231291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.232677 (+ 1386us) server_negotiation.cc:407] Connection header received
0807 06:48:16.234808 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.234813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.234887 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.234984 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.236766 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.237609 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.239402 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.239902 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.240009 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.244171 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:16.244209 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:16.244217 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:16.244289 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:16.247904 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.248783 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.248788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.248790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.248934 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.249360 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.249363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.249365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.249645 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:16.249763 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.250545 (+ 782us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.250797 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":234,"threads_started":1}
I20260807 06:48:16.252112 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.231485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:32771 (local address 127.27.54.67:38453)
0807 06:48:16.232491 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:16.232551 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:16.234452 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:16.235221 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:16.235229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:16.235926 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:16.236565 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.236585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.237832 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.237836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.239256 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.239271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.240112 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.240117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.240333 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.241107 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:16.241131 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:16.243804 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:16.248066 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:16.248074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:16.248091 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:16.248658 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:16.249029 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:16.249033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:16.249035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:16.249251 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:16.249802 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:16.249808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:16.250183 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 06:48:16.250436 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 06:48:16.250940 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":441,"threads_started":1}
I20260807 06:48:16.253650 29303 heartbeater.cc:347] Connected to a master server at 127.27.54.126:32771
I20260807 06:48:16.253989 29303 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:16.254891 29303 heartbeater.cc:511] Master 127.27.54.126:32771 requested a full tablet report, sending...
I20260807 06:48:16.256381 28662 ts_manager.cc:194] Registered new tserver with Master: c1ff87b51f1d40d6a6cbf0728ed0623b (127.27.54.67:40717)
I20260807 06:48:16.257462 28662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:38453
I20260807 06:48:16.259429 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:48:16.280485 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.267340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.27.54.126:32771)
0807 06:48:16.267587 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.267597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.268131 (+ 534us) server_negotiation.cc:407] Connection header received
0807 06:48:16.268289 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.268294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.268366 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.268458 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.270068 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.270854 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.272130 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.272519 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.272616 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.275156 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:16.275162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:16.275165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:16.275200 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:16.277685 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.278355 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.278359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.278361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.278489 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.278963 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.278968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.278970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.279333 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:16.279559 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.279933 (+ 374us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.280122 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:48:16.289778 28662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:16.298228 28662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:16.340691 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.317333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:40717 (local address 127.0.0.1:51174)
0807 06:48:16.329120 (+ 11787us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:16.329147 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:16.329348 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:16.332257 (+ 2909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:16.332262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:16.332291 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:16.332748 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.332764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.335019 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.335024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.337364 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.337380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.339733 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.339741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.339996 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.340030 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:48:16.340139 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 06:48:16.340293 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11583,"thread_start_us":11407,"threads_started":1}
I20260807 06:48:16.341279 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.316676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:36233 (local address 127.0.0.1:59110)
0807 06:48:16.326026 (+ 9350us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:16.326097 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:16.326525 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:16.329914 (+ 3389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:16.329922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:16.329956 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:16.330366 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.330380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.333673 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.333679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.336512 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.336530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.338823 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.338832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.339076 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.339162 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 06:48:16.339365 (+ 203us) client_negotiation.cc:240] Negotiation successful
0807 06:48:16.340172 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8932,"mutex_wait_us":1,"thread_start_us":8728,"threads_started":1}
I20260807 06:48:16.341851 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.317700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.27.54.67:40717)
0807 06:48:16.331152 (+ 13452us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.331162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.331206 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:48:16.331313 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.331322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.331638 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.331877 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:16.333029 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.334782 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.337573 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.339163 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.339315 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.339580 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.341289 (+ 1709us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.341469 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13230,"thread_start_us":13030,"threads_started":1}
I20260807 06:48:16.344386 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.318536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.27.54.65:36233)
0807 06:48:16.328789 (+ 10253us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.328799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.328836 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:48:16.328965 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.328975 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.329287 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.329561 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:16.331521 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.333475 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.336782 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.338447 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.338621 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.338978 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.339575 (+ 597us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.339804 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9997,"thread_start_us":9771,"threads_started":1}
I20260807 06:48:16.347683 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.331159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:36311 (local address 127.0.0.1:40838)
0807 06:48:16.333010 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:16.333034 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:16.333180 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:16.334740 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:16.334745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:16.334771 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:16.339329 (+ 4558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.339345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.341890 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.341896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.343894 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.343908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.346621 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.346629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.346892 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.346948 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 06:48:16.347050 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 06:48:16.347238 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1698,"thread_start_us":4641,"threads_started":1}
I20260807 06:48:16.347767 29205 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4bf39294ff4f838fc19b470436ee10 (DEFAULT_TABLE table=test-table [id=50c494921a604c4ea4578307ccbcdd13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:16.348615 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.327289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.27.54.66:36311)
0807 06:48:16.331412 (+ 4123us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.331420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.333407 (+ 1987us) server_negotiation.cc:407] Connection header received
0807 06:48:16.333602 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.333614 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.334100 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.334604 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:16.340010 (+ 5406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.341655 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.344071 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.345981 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.346193 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.346456 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.347981 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.348215 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3934,"thread_start_us":4006,"threads_started":1}
I20260807 06:48:16.348629 28828 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4bf39294ff4f838fc19b470436ee10 (DEFAULT_TABLE table=test-table [id=50c494921a604c4ea4578307ccbcdd13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:16.349844 29205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4bf39294ff4f838fc19b470436ee10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:16.354863 28828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4bf39294ff4f838fc19b470436ee10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:16.354694 29011 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4bf39294ff4f838fc19b470436ee10 (DEFAULT_TABLE table=test-table [id=50c494921a604c4ea4578307ccbcdd13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:16.356261 29011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4bf39294ff4f838fc19b470436ee10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:16.369413 29329 tablet_bootstrap.cc:492] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: Bootstrap starting.
I20260807 06:48:16.372534 29330 tablet_bootstrap.cc:492] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: Bootstrap starting.
I20260807 06:48:16.373894 29332 tablet_bootstrap.cc:492] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: Bootstrap starting.
I20260807 06:48:16.373981 29329 tablet_bootstrap.cc:654] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:16.376127 29329 log.cc:824] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:16.377372 29330 tablet_bootstrap.cc:654] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:16.377684 29332 tablet_bootstrap.cc:654] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:16.378895 29330 log.cc:824] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:16.378988 29332 log.cc:824] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:16.383694 29330 tablet_bootstrap.cc:492] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: No bootstrap required, opened a new log
I20260807 06:48:16.383728 29329 tablet_bootstrap.cc:492] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: No bootstrap required, opened a new log
I20260807 06:48:16.384069 29329 ts_tablet_manager.cc:1397] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20260807 06:48:16.384109 29330 ts_tablet_manager.cc:1397] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260807 06:48:16.384732 29332 tablet_bootstrap.cc:492] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: No bootstrap required, opened a new log
I20260807 06:48:16.385021 29332 ts_tablet_manager.cc:1397] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260807 06:48:16.397290 29330 raft_consensus.cc:348] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.397279 29332 raft_consensus.cc:348] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.397946 29332 raft_consensus.cc:374] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:16.397662 29329 raft_consensus.cc:348] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.398197 29330 raft_consensus.cc:374] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:16.398125 29332 raft_consensus.cc:729] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfd5868f2704d2985274259b006b58d, State: Initialized, Role: FOLLOWER
I20260807 06:48:16.398209 29329 raft_consensus.cc:374] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:16.398321 29329 raft_consensus.cc:729] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ff87b51f1d40d6a6cbf0728ed0623b, State: Initialized, Role: FOLLOWER
I20260807 06:48:16.398379 29330 raft_consensus.cc:729] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd01238ed8b42889e889d630b548e87, State: Initialized, Role: FOLLOWER
I20260807 06:48:16.399279 29329 consensus_queue.cc:260] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.399331 29332 consensus_queue.cc:260] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.400936 29303 heartbeater.cc:503] Master 127.27.54.126:32771 was elected leader, sending a full tablet report...
I20260807 06:48:16.401590 29329 ts_tablet_manager.cc:1428] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260807 06:48:16.403302 29330 consensus_queue.cc:260] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.404498 29104 heartbeater.cc:503] Master 127.27.54.126:32771 was elected leader, sending a full tablet report...
I20260807 06:48:16.405260 29330 ts_tablet_manager.cc:1428] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: Time spent starting tablet: real 0.021s user 0.013s sys 0.000s
I20260807 06:48:16.406446 29332 ts_tablet_manager.cc:1428] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: Time spent starting tablet: real 0.021s user 0.008s sys 0.010s
W20260807 06:48:16.427790 29105 tablet.cc:2369] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:48:16.479482 29304 tablet.cc:2369] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:16.623270 29341 raft_consensus.cc:482] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:16.623677 29341 raft_consensus.cc:504] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.625844 29341 leader_election.cc:288] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdfd5868f2704d2985274259b006b58d (127.27.54.65:36233), c1ff87b51f1d40d6a6cbf0728ed0623b (127.27.54.67:40717)
W20260807 06:48:16.632167 28914 tablet.cc:2369] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:16.635771 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.626488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:35135 (local address 127.27.54.67:40717)
0807 06:48:16.626744 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.626752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.627602 (+ 850us) server_negotiation.cc:407] Connection header received
0807 06:48:16.627819 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.627825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.627911 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.628023 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:16.628795 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.629932 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.633386 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.634265 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.634496 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.634576 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.635346 (+ 770us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.635507 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 06:48:16.636005 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.626407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:50929 (local address 127.27.54.65:36233)
0807 06:48:16.626635 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.626642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.627602 (+ 960us) server_negotiation.cc:407] Connection header received
0807 06:48:16.627834 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.627839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.627913 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.628023 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:16.628795 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.629932 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.633384 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.634703 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.634861 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.634984 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.635602 (+ 618us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.635740 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:48:16.636162 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.626260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:36233 (local address 127.27.54.66:50929)
0807 06:48:16.627483 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:16.627500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:16.627646 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:16.628056 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:16.628061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:16.628089 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:16.628606 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.628626 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.630094 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.630100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.633137 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.633159 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.635225 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.635230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.635388 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.635410 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:48:16.635570 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 06:48:16.635716 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":222,"thread_start_us":617,"threads_started":1}
I20260807 06:48:16.636224 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.626178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:40717 (local address 127.27.54.66:35135)
0807 06:48:16.627417 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:16.627446 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:16.627676 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:16.628056 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:16.628061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:16.628103 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:16.628652 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.628667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.630096 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.630100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.633137 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:16.633159 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.634996 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.635004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:16.635157 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.635189 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:48:16.635324 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 06:48:16.635484 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":186,"spinlock_wait_cycles":60032,"thread_start_us":855,"threads_started":1}
I20260807 06:48:16.637462 28853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4bf39294ff4f838fc19b470436ee10" candidate_uuid: "0bd01238ed8b42889e889d630b548e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdfd5868f2704d2985274259b006b58d" is_pre_election: true
I20260807 06:48:16.638159 28853 raft_consensus.cc:2454] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bd01238ed8b42889e889d630b548e87 in term 0.
I20260807 06:48:16.639072 28945 leader_election.cc:302] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd01238ed8b42889e889d630b548e87, bdfd5868f2704d2985274259b006b58d; no voters:
I20260807 06:48:16.638995 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4bf39294ff4f838fc19b470436ee10" candidate_uuid: "0bd01238ed8b42889e889d630b548e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" is_pre_election: true
I20260807 06:48:16.639597 29232 raft_consensus.cc:2454] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bd01238ed8b42889e889d630b548e87 in term 0.
I20260807 06:48:16.639616 29341 raft_consensus.cc:2781] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:16.639799 29341 raft_consensus.cc:482] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:16.639923 29341 raft_consensus.cc:3037] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:16.642483 29341 raft_consensus.cc:504] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.643172 29341 leader_election.cc:288] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [CANDIDATE]: Term 1 election: Requested vote from peers bdfd5868f2704d2985274259b006b58d (127.27.54.65:36233), c1ff87b51f1d40d6a6cbf0728ed0623b (127.27.54.67:40717)
I20260807 06:48:16.643776 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4bf39294ff4f838fc19b470436ee10" candidate_uuid: "0bd01238ed8b42889e889d630b548e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b"
I20260807 06:48:16.643813 28853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4bf39294ff4f838fc19b470436ee10" candidate_uuid: "0bd01238ed8b42889e889d630b548e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdfd5868f2704d2985274259b006b58d"
I20260807 06:48:16.644001 29232 raft_consensus.cc:3037] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:16.644054 28853 raft_consensus.cc:3037] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:16.646435 29232 raft_consensus.cc:2454] T 1b4bf39294ff4f838fc19b470436ee10 P c1ff87b51f1d40d6a6cbf0728ed0623b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bd01238ed8b42889e889d630b548e87 in term 1.
I20260807 06:48:16.646466 28853 raft_consensus.cc:2454] T 1b4bf39294ff4f838fc19b470436ee10 P bdfd5868f2704d2985274259b006b58d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bd01238ed8b42889e889d630b548e87 in term 1.
I20260807 06:48:16.647030 28949 leader_election.cc:302] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd01238ed8b42889e889d630b548e87, c1ff87b51f1d40d6a6cbf0728ed0623b; no voters:
I20260807 06:48:16.647382 29341 raft_consensus.cc:2781] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:16.647873 29341 raft_consensus.cc:686] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [term 1 LEADER]: Becoming Leader. State: Replica: 0bd01238ed8b42889e889d630b548e87, State: Running, Role: LEADER
I20260807 06:48:16.648387 29341 consensus_queue.cc:237] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } }
I20260807 06:48:16.654548 28662 catalog_manager.cc:5495] T 1b4bf39294ff4f838fc19b470436ee10 P 0bd01238ed8b42889e889d630b548e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bd01238ed8b42889e889d630b548e87 (127.27.54.66). New cstate: current_term: 1 leader_uuid: "0bd01238ed8b42889e889d630b548e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfd5868f2704d2985274259b006b58d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bd01238ed8b42889e889d630b548e87" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1ff87b51f1d40d6a6cbf0728ed0623b" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 40717 } health_report { overall_health: UNKNOWN } } }
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:16.724215 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.708105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.27.54.65:36233)
0807 06:48:16.708357 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.708366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.708922 (+ 556us) server_negotiation.cc:407] Connection header received
0807 06:48:16.709072 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.709076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.709149 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.709240 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.710904 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.711681 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.712982 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.713469 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.713587 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.718049 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:16.718066 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:16.718083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:16.718132 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:16.721375 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.722097 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.722105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.722107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.722282 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.722618 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.722620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.722622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.723076 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:16.723366 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.723624 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.723814 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:16.788385 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.772160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.27.54.65:36233)
0807 06:48:16.772444 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.772451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.772757 (+ 306us) server_negotiation.cc:407] Connection header received
0807 06:48:16.772860 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.772864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.772942 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.773044 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.774868 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.775648 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.776994 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.777432 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.777541 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.782484 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:16.782491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:16.782494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:16.782540 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:16.785561 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.786355 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.786360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.786362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.786538 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.787008 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.787012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.787014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.787338 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:16.787572 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.787800 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.787999 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:16.847796 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.831583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.27.54.65:36233)
0807 06:48:16.831902 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.831910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.832226 (+ 316us) server_negotiation.cc:407] Connection header received
0807 06:48:16.832469 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.832473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.832544 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.832662 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.834131 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.835069 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.836547 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.837023 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.837175 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.841808 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:16.841822 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:16.841841 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:16.841911 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:16.844786 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.845610 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.845616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.845620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.845763 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.846174 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.846180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.846187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.846524 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:16.846851 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.847154 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.847424 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260807 06:48:16.850222 28801 tablet_service.cc:2908] Rejecting scan request for tablet 1b4bf39294ff4f838fc19b470436ee10: Uninitialized: clean time has not yet been initialized
I20260807 06:48:16.853952 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28746
I20260807 06:48:16.868937 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.855011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.27.54.65:36233)
0807 06:48:16.855228 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.855236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.856037 (+ 801us) server_negotiation.cc:407] Connection header received
0807 06:48:16.856242 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.856247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.856336 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.856432 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.857779 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.858491 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.859928 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.860390 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.860509 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:16.863187 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:16.863193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:16.863196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:16.863234 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:16.866016 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.866985 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.866990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.866992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.867136 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:16.867557 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:16.867561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:16.867563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:16.867885 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:16.868115 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:16.868385 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 06:48:16.868576 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:48:16.869758 28904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:16.962886 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28746
I20260807 06:48:16.991477 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28922
Aug 07 06:48:16 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:17.007575 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:16.992313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.27.54.66:36311)
0807 06:48:16.992869 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:16.992874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:16.992892 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:48:16.992950 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:16.992954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:16.993024 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:16.993128 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:16.994371 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.995194 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.996546 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:16.996948 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:16.997052 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:17.001665 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:17.001683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:17.001699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:17.001739 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:17.004788 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:17.005519 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:17.005526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:17.005528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:17.005678 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:17.006022 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:17.006025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:17.006027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:17.006395 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:17.006631 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:17.007074 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 06:48:17.007264 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":299,"threads_started":1}
I20260807 06:48:17.008277 29097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:17.104998 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28922
I20260807 06:48:17.130995 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29115
Aug 07 06:48:17 dist-test-slave-3kk6 krb5kdc[28606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:17.151544 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:17.132086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.27.54.67:40717)
0807 06:48:17.132266 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:17.132271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:17.132351 (+ 80us) server_negotiation.cc:407] Connection header received
0807 06:48:17.132558 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:17.132562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:17.132630 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:17.132716 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:17.134901 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:17.135723 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:17.138095 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:17.138509 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:17.138618 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:17.145016 (+ 6398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:17.145037 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:17.145053 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:17.145103 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:17.148297 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:17.149203 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:17.149212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:17.149214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:17.149384 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:17.149799 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:17.149810 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:17.149813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:17.150209 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:17.150535 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:17.150926 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 06:48:17.151167 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:48:17.152205 29294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:17.244005 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 29115
I20260807 06:48:17.268903 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28623
I20260807 06:48:17.270524 28700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:17.383744 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28623
2026-08-07T06:48:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4892 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:17 dist-test-slave-3kk6 krb5kdc[29369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:17 dist-test-slave-3kk6 krb5kdc[29369](info): set up 2 sockets
Aug 07 06:48:17 dist-test-slave-3kk6 krb5kdc[29369](info): commencing operation
krb5kdc: starting...
W20260807 06:48:18.797801 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260807 06:48:18.994560 27865 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.197s user 0.000s sys 0.008s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260807 06:48:19.194151 27865 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.199s user 0.004s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260807 06:48:19.345428 27865 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.151s user 0.004s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/test-user.keytab.
W20260807 06:48:19.493518 27865 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.148s user 0.000s sys 0.008s
Aug 07 06:48:19 dist-test-slave-3kk6 krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:19.607090 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39259
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:39259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:19.960112 29389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:19.960484 29389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:19.960670 29389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:19.968209 29389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:19.968351 29389 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 06:48:19.968439 29389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:19.968523 29389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:19.968590 29389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:19.968677 29389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:19.979269 29389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:39259
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:19.980687 29389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:19.982087 29389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:19.994939 29397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:19.994872 29398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:19.994916 29402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:19.996183 29389 server_base.cc:1034] running on GCE node
I20260807 06:48:19.997016 29389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:19.998541 29389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:19.999773 29389 hybrid_clock.cc:648] HybridClock initialized: now 1786085299999699 us; error 48 us; skew 500 ppm
I20260807 06:48:20.000090 29389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:19 dist-test-slave-3kk6 krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:20.003360 29389 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:20.004953 29389 webserver.cc:460] Webserver started at http://127.27.54.126:37221/ using document root <none> and password file <none>
I20260807 06:48:20.005676 29389 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:20.005769 29389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:20.006088 29389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:20.009281 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "c9bab06eaeab4ba88d7f59a279aa5df8"
format_stamp: "Formatted at 2026-08-07 06:48:19 on dist-test-slave-3kk6"
server_key: "d0ca766f92466c802905b816b7901da5"
server_key_iv: "a69d37d472b64687758adde19e3d8bc1"
server_key_version: "encryptionkey@0"
I20260807 06:48:20.010128 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "c9bab06eaeab4ba88d7f59a279aa5df8"
format_stamp: "Formatted at 2026-08-07 06:48:19 on dist-test-slave-3kk6"
server_key: "d0ca766f92466c802905b816b7901da5"
server_key_iv: "a69d37d472b64687758adde19e3d8bc1"
server_key_version: "encryptionkey@0"
I20260807 06:48:20.015656 29389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 06:48:20.019158 29412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:20.020561 29389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 06:48:20.020750 29389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "c9bab06eaeab4ba88d7f59a279aa5df8"
format_stamp: "Formatted at 2026-08-07 06:48:19 on dist-test-slave-3kk6"
server_key: "d0ca766f92466c802905b816b7901da5"
server_key_iv: "a69d37d472b64687758adde19e3d8bc1"
server_key_version: "encryptionkey@0"
I20260807 06:48:20.020980 29389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:20.050530 29389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:20.055070 29389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:20.055341 29389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:20.078394 29389 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:39259
I20260807 06:48:20.078416 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:39259 every 8 connection(s)
I20260807 06:48:20.079957 29389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:20.081465 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 29389
I20260807 06:48:20.081691 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:20.082388 27865 external_mini_cluster.cc:1428] Setting key fae05c45b86c46aa032f923c9dba378f
I20260807 06:48:20.084717 29505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:48:20 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:20.099993 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: Bootstrap starting.
I20260807 06:48:20.104400 29505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:20.105530 29505 log.cc:824] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:20.108995 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: No bootstrap required, opened a new log
I20260807 06:48:20.109025 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:20.084538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.27.54.126:39259)
0807 06:48:20.085420 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:20.085432 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:20.085476 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:48:20.087278 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:20.087301 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:20.087720 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:20.088096 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:20.090054 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.091517 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.093376 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.094099 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.094256 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:20.100709 (+ 6453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:20.100723 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:20.100746 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:20.100810 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:20.104636 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:20.105426 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:20.105436 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:20.105444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:20.105628 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:20.106077 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:20.106082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:20.106086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:20.106720 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:20.107058 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:20.107532 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 06:48:20.107988 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":459,"threads_started":1}
I20260807 06:48:20.117776 29505 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39259 } }
I20260807 06:48:20.118285 29505 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:20.118428 29505 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bab06eaeab4ba88d7f59a279aa5df8, State: Initialized, Role: FOLLOWER
I20260807 06:48:20.119171 29505 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39259 } }
I20260807 06:48:20.119433 29505 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:20.119583 29505 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:20.119778 29505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:20.122076 29505 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39259 } }
I20260807 06:48:20.122627 29505 leader_election.cc:302] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9bab06eaeab4ba88d7f59a279aa5df8; no voters:
I20260807 06:48:20.123381 29505 leader_election.cc:288] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:20.123523 29514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:20.124323 29514 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [term 1 LEADER]: Becoming Leader. State: Replica: c9bab06eaeab4ba88d7f59a279aa5df8, State: Running, Role: LEADER
I20260807 06:48:20.124958 29514 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39259 } }
I20260807 06:48:20.125938 29505 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:20.129501 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39259 } } }
I20260807 06:48:20.130007 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:20.130875 29516 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9bab06eaeab4ba88d7f59a279aa5df8. Latest consensus state: current_term: 1 leader_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bab06eaeab4ba88d7f59a279aa5df8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39259 } } }
I20260807 06:48:20.131251 29516 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:20.131975 29521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:20.137792 29521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:20.147940 29521 catalog_manager.cc:1347] Generated new cluster ID: 9d822fc4a6a5406fbf4a4cbb93aa709b
I20260807 06:48:20.148089 29521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:20.163092 29521 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:20.164525 29521 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:20.178413 29521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: Generated new TSK 0
I20260807 06:48:20.179382 29521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:20.241842 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39259
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:20.596072 29544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:20.596397 29544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:20.596503 29544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:20.604096 29544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:20.604223 29544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:20.604311 29544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:48:20.604398 29544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:20.614925 29544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:39259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:20.616246 29544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:20.617574 29544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:20.631173 29551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:20.631547 29556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:20.631824 29553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:20.632565 29544 server_base.cc:1034] running on GCE node
I20260807 06:48:20.633244 29544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:20.634205 29544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:20.635478 29544 hybrid_clock.cc:648] HybridClock initialized: now 1786085300635390 us; error 67 us; skew 500 ppm
I20260807 06:48:20.635843 29544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:20 dist-test-slave-3kk6 krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:20.639321 29544 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:20.640852 29544 webserver.cc:460] Webserver started at http://127.27.54.65:35933/ using document root <none> and password file <none>
I20260807 06:48:20.641605 29544 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:20.641742 29544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:20.642098 29544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:20.645359 29544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "e20d5944df6649539e2c25443a74e9b3"
format_stamp: "Formatted at 2026-08-07 06:48:20 on dist-test-slave-3kk6"
server_key: "d308ec1640c96762b0fa236f27e7deda"
server_key_iv: "1384d70c8b73154237b07cb368effe73"
server_key_version: "encryptionkey@0"
I20260807 06:48:20.646317 29544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "e20d5944df6649539e2c25443a74e9b3"
format_stamp: "Formatted at 2026-08-07 06:48:20 on dist-test-slave-3kk6"
server_key: "d308ec1640c96762b0fa236f27e7deda"
server_key_iv: "1384d70c8b73154237b07cb368effe73"
server_key_version: "encryptionkey@0"
I20260807 06:48:20.652015 29544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:48:20.655772 29569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:20.657133 29544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:20.657327 29544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "e20d5944df6649539e2c25443a74e9b3"
format_stamp: "Formatted at 2026-08-07 06:48:20 on dist-test-slave-3kk6"
server_key: "d308ec1640c96762b0fa236f27e7deda"
server_key_iv: "1384d70c8b73154237b07cb368effe73"
server_key_version: "encryptionkey@0"
I20260807 06:48:20.657584 29544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:20.669567 29544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:20.674505 29544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:20.674883 29544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:20.677976 29544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:20.678097 29544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:20.678279 29544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:20.678381 29544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:48:20 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:20.707970 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:20.678896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:55957 (local address 127.27.54.126:39259)
0807 06:48:20.682350 (+ 3454us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:20.682355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:20.682379 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:48:20.684979 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:20.684986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:20.685087 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:20.685189 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:20.688289 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.689615 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.693282 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.693732 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.693855 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:20.699944 (+ 6089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:20.699950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:20.699954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:20.699995 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:20.702941 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:20.704259 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:20.704268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:20.704274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:20.704447 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:20.705148 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:20.705153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:20.705156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:20.705575 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:20.706816 (+ 1241us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:20.707041 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 06:48:20.707390 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3327,"thread_start_us":3224,"threads_started":1}
I20260807 06:48:20.708567 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:20.679891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.27.54.65:55957)
0807 06:48:20.681088 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:20.681154 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:20.684458 (+ 3304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:20.685567 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:20.685584 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:20.686634 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:20.687941 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:20.687982 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.689926 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.689932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:20.693063 (+ 3131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:20.693087 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.694026 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.694037 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:20.694440 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:20.695533 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:20.695574 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:20.699617 (+ 4043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:20.703164 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:20.703176 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:20.703192 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:20.704029 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:20.704590 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:20.704595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:20.704599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:20.704964 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:20.705851 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:20.705864 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:20.706440 (+ 576us) client_negotiation.cc:769] Sending connection context
0807 06:48:20.706852 (+ 412us) client_negotiation.cc:240] Negotiation successful
0807 06:48:20.707342 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":545,"threads_started":1}
I20260807 06:48:20.736392 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:20.715353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:41445 (local address 127.27.54.126:39259)
0807 06:48:20.715537 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:20.715542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:20.719437 (+ 3895us) server_negotiation.cc:407] Connection header received
0807 06:48:20.719657 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:20.719661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:20.719732 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:20.719822 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:20.721349 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.722059 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.724871 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.725828 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.725973 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:20.728495 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:20.728505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:20.728512 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:20.728569 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:20.732894 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:20.733647 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:20.733656 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:20.733661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:20.733882 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:20.734392 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:20.734398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:20.734401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:20.734870 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:20.735062 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:20.735693 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 06:48:20.735966 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:48:20.736584 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:20.715399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.27.54.65:41445)
0807 06:48:20.719279 (+ 3880us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:20.719308 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:20.719523 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:20.719892 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:20.719900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:20.720659 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:20.721171 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:20.721189 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.722224 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.722232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:20.724611 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:20.724641 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:20.726021 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:20.726026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:20.726231 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:20.727052 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:20.727070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:20.728257 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:20.733054 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:20.733058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:20.733061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:20.733495 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:20.734032 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:20.734040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:20.734049 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:20.734250 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:20.735133 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:20.735140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:20.735418 (+ 278us) client_negotiation.cc:769] Sending connection context
0807 06:48:20.735575 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 06:48:20.735999 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3665,"thread_start_us":3244,"threads_started":1}
I20260807 06:48:20.738731 29544 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:44445
I20260807 06:48:20.738765 29744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:44445 every 8 connection(s)
I20260807 06:48:20.740074 29544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:20.742868 29746 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39259
I20260807 06:48:20.743125 29746 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:20.743825 29746 heartbeater.cc:511] Master 127.27.54.126:39259 requested a full tablet report, sending...
I20260807 06:48:20.745908 29439 ts_manager.cc:194] Registered new tserver with Master: e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445)
I20260807 06:48:20.747258 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 29544
I20260807 06:48:20.747486 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:20.748085 29439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:55957
I20260807 06:48:20.748179 27865 external_mini_cluster.cc:1428] Setting key f922c63c6ae34d489ad009450dcdf4f0
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:20.805442 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39259
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:21.156216 29751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:21.156523 29751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:21.156626 29751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:21.164490 29751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:21.164626 29751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:21.164714 29751 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:48:21.164813 29751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:21.175346 29751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:39259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:21.176735 29751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:21.178179 29751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:21.191978 29758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:21.192044 29759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:21.192283 29761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:21.192822 29751 server_base.cc:1034] running on GCE node
I20260807 06:48:21.193493 29751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:21.194495 29751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:21.195765 29751 hybrid_clock.cc:648] HybridClock initialized: now 1786085301195699 us; error 67 us; skew 500 ppm
I20260807 06:48:21.196101 29751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:21 dist-test-slave-3kk6 krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:21.199959 29751 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:21.201807 29751 webserver.cc:460] Webserver started at http://127.27.54.66:38335/ using document root <none> and password file <none>
I20260807 06:48:21.202634 29751 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:21.202869 29751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:21.203246 29751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:21.206703 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e"
format_stamp: "Formatted at 2026-08-07 06:48:21 on dist-test-slave-3kk6"
server_key: "f869290e749b752f846e90def4568370"
server_key_iv: "3dcda4c05ec80d2f2e5a03b047ec210f"
server_key_version: "encryptionkey@0"
I20260807 06:48:21.207708 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e"
format_stamp: "Formatted at 2026-08-07 06:48:21 on dist-test-slave-3kk6"
server_key: "f869290e749b752f846e90def4568370"
server_key_iv: "3dcda4c05ec80d2f2e5a03b047ec210f"
server_key_version: "encryptionkey@0"
I20260807 06:48:21.213825 29751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260807 06:48:21.218245 29773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:21.219969 29751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:21.220134 29751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e"
format_stamp: "Formatted at 2026-08-07 06:48:21 on dist-test-slave-3kk6"
server_key: "f869290e749b752f846e90def4568370"
server_key_iv: "3dcda4c05ec80d2f2e5a03b047ec210f"
server_key_version: "encryptionkey@0"
I20260807 06:48:21.220366 29751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:21.250496 29751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:21.255080 29751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:21.255353 29751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:21.258284 29751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:21.258384 29751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:21.258576 29751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:21.258683 29751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:48:21 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:21.285151 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.258896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:51041 (local address 127.27.54.126:39259)
0807 06:48:21.259400 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:21.259407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:21.260710 (+ 1303us) server_negotiation.cc:407] Connection header received
0807 06:48:21.263958 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:21.263967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:21.264075 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:21.264233 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:21.267146 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.268231 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.270997 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.271679 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.271821 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.276224 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:21.276234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:21.276241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:21.276298 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:21.280184 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.281736 (+ 1552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.281744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.281750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.281956 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.282670 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.282678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.282683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.283200 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:21.283443 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:21.284203 (+ 760us) server_negotiation.cc:299] Negotiation successful
0807 06:48:21.284564 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260807 06:48:21.285797 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.259550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.27.54.66:51041)
0807 06:48:21.260485 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:21.260560 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:21.263577 (+ 3017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:21.264551 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:21.264567 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:21.265658 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:21.266869 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.266912 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.268463 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.268469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.270823 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.270837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.271962 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.271973 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.272191 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.273023 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:21.273058 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:21.275888 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:21.280497 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.280513 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.280542 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.281531 (+ 989us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.282085 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.282091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.282095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.282460 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.283375 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:21.283383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:21.283775 (+ 392us) client_negotiation.cc:769] Sending connection context
0807 06:48:21.284034 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 06:48:21.284510 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":347,"threads_started":1}
I20260807 06:48:21.311014 29751 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:46203
I20260807 06:48:21.311062 29946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:46203 every 8 connection(s)
I20260807 06:48:21.312814 29751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:21.314746 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.295234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:59821 (local address 127.27.54.126:39259)
0807 06:48:21.295473 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:21.295482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:21.296389 (+ 907us) server_negotiation.cc:407] Connection header received
0807 06:48:21.296543 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:21.296552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:21.296673 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:21.296821 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:21.299051 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.300427 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.302995 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.303830 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.303965 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.307477 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:21.307486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:21.307492 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:21.307544 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:21.311313 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.312227 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.312232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.312234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.312363 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.312777 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.312780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.312782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.313161 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:21.313280 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:21.314009 (+ 729us) server_negotiation.cc:299] Negotiation successful
0807 06:48:21.314363 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 06:48:21.315267 29947 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39259
I20260807 06:48:21.315415 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.295261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.27.54.66:59821)
0807 06:48:21.296238 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:21.296262 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:21.296424 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:21.297052 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:21.297060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:21.298052 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:21.298751 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.298784 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.300643 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.300647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.302736 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.302761 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.304384 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.304394 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.304649 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.305559 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:21.305597 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:21.307205 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:21.311461 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.311466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.311470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.312097 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.312477 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.312482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.312485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.312647 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.313291 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:21.313298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:21.313603 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 06:48:21.314527 (+ 924us) client_negotiation.cc:240] Negotiation successful
0807 06:48:21.314872 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":641,"threads_started":1}
I20260807 06:48:21.315618 29947 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:21.316330 29947 heartbeater.cc:511] Master 127.27.54.126:39259 requested a full tablet report, sending...
I20260807 06:48:21.317724 29439 ts_manager.cc:194] Registered new tserver with Master: 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203)
I20260807 06:48:21.318823 29439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:51041
I20260807 06:48:21.320393 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 29751
I20260807 06:48:21.320581 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:21.321044 27865 external_mini_cluster.cc:1428] Setting key d24303245eb15f05ae44baf4de7ca95a
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:21.374018 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39259
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:21.725201 29953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:21.725589 29953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:21.725782 29953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:21.733230 29953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:21.733405 29953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:21.733546 29953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:48:21.733678 29953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:21.744181 29953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:39259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:21.745503 29953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:21.746999 29953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:48:21.751653 29746 heartbeater.cc:503] Master 127.27.54.126:39259 was elected leader, sending a full tablet report...
W20260807 06:48:21.762240 29960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:21.762215 29961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:21.762215 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:21.763881 29953 server_base.cc:1034] running on GCE node
I20260807 06:48:21.764627 29953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:21.765686 29953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:21.766939 29953 hybrid_clock.cc:648] HybridClock initialized: now 1786085301766873 us; error 47 us; skew 500 ppm
I20260807 06:48:21.767280 29953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:21 dist-test-slave-3kk6 krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:21.770869 29953 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:21.772578 29953 webserver.cc:460] Webserver started at http://127.27.54.67:41625/ using document root <none> and password file <none>
I20260807 06:48:21.773327 29953 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:21.773432 29953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:21.773743 29953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:21.777050 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "55ca5e28384643bf810e478b54e51afd"
format_stamp: "Formatted at 2026-08-07 06:48:21 on dist-test-slave-3kk6"
server_key: "d4cc5366ba362142de0dc870879f9ca3"
server_key_iv: "ebd8fdf0100e1037b7ea970da51c8873"
server_key_version: "encryptionkey@0"
I20260807 06:48:21.777921 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "55ca5e28384643bf810e478b54e51afd"
format_stamp: "Formatted at 2026-08-07 06:48:21 on dist-test-slave-3kk6"
server_key: "d4cc5366ba362142de0dc870879f9ca3"
server_key_iv: "ebd8fdf0100e1037b7ea970da51c8873"
server_key_version: "encryptionkey@0"
I20260807 06:48:21.783804 29953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 06:48:21.788419 29975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:21.790429 29953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:48:21.790680 29953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "55ca5e28384643bf810e478b54e51afd"
format_stamp: "Formatted at 2026-08-07 06:48:21 on dist-test-slave-3kk6"
server_key: "d4cc5366ba362142de0dc870879f9ca3"
server_key_iv: "ebd8fdf0100e1037b7ea970da51c8873"
server_key_version: "encryptionkey@0"
I20260807 06:48:21.790964 29953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:21.814424 29953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:21.819756 29953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:21.820086 29953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:21.823150 29953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:21.823272 29953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:21.823462 29953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:21.823566 29953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:48:21 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:21.853513 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.824658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:53477 (local address 127.27.54.126:39259)
0807 06:48:21.825320 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:21.825325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:21.826917 (+ 1592us) server_negotiation.cc:407] Connection header received
0807 06:48:21.829816 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:21.829827 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:21.829931 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:21.830054 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:21.833346 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.834247 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.836874 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.837364 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.837530 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.843627 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:21.843639 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:21.843647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:21.843710 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:21.848564 (+ 4854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.850192 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.850198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.850202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.850369 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.851219 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.851229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.851234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.851686 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:21.851876 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:21.852780 (+ 904us) server_negotiation.cc:299] Negotiation successful
0807 06:48:21.853055 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":408,"threads_started":1}
I20260807 06:48:21.854743 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.825059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.27.54.67:53477)
0807 06:48:21.826675 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:21.826745 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:21.829293 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:21.830821 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:21.830840 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:21.831889 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:21.833085 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.833122 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.834444 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.834452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.836676 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.836693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.837680 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.837692 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.838103 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.839206 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:21.839241 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:21.843216 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:21.848857 (+ 5641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.848872 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.848906 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.849993 (+ 1087us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.850583 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.850588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.850596 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.851051 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.851851 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:21.851859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:21.852324 (+ 465us) client_negotiation.cc:769] Sending connection context
0807 06:48:21.852645 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 06:48:21.853303 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":568,"threads_started":1}
I20260807 06:48:21.880496 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.861375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:51499 (local address 127.27.54.126:39259)
0807 06:48:21.861558 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:21.861563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:21.863004 (+ 1441us) server_negotiation.cc:407] Connection header received
0807 06:48:21.863256 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:21.863265 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:21.863379 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:21.863494 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:21.865239 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.866042 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.868916 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.869463 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.869637 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.872852 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:21.872858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:21.872861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:21.872898 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:21.876622 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.877706 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.877714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.877719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.877907 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.878455 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.878460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.878463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.878936 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:21.879115 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:21.879859 (+ 744us) server_negotiation.cc:299] Negotiation successful
0807 06:48:21.880107 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:48:21.880681 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.861815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.27.54.67:51499)
0807 06:48:21.862878 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:21.862897 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:21.863090 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:21.863606 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:21.863614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:21.864474 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:21.865007 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.865027 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.866203 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.866216 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.868688 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:21.868716 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.869740 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.869756 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:21.870040 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.871088 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:21.871123 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:21.872616 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:21.876799 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.876805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.876809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.877478 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.878035 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:21.878041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:21.878044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:21.878275 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:21.879281 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:21.879288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:21.879555 (+ 267us) client_negotiation.cc:769] Sending connection context
0807 06:48:21.879748 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 06:48:21.880142 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"thread_start_us":600,"threads_started":1}
I20260807 06:48:21.887692 30150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:39983 every 8 connection(s)
I20260807 06:48:21.887706 29953 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:39983
I20260807 06:48:21.889115 29953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:21.889720 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 29953
I20260807 06:48:21.889912 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:21.890530 27865 external_mini_cluster.cc:1428] Setting key fee6794c901c0b68f427e25aadb5b689
I20260807 06:48:21.891642 30151 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39259
I20260807 06:48:21.891929 30151 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:21.892701 30151 heartbeater.cc:511] Master 127.27.54.126:39259 requested a full tablet report, sending...
I20260807 06:48:21.894114 29439 ts_manager.cc:194] Registered new tserver with Master: 55ca5e28384643bf810e478b54e51afd (127.27.54.67:39983)
I20260807 06:48:21.895049 29439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:53477
I20260807 06:48:21.906219 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:48:21 dist-test-slave-3kk6 krb5kdc[29369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:48:21 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:21.964788 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.948918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.27.54.126:39259)
0807 06:48:21.949172 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:21.949181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:21.949523 (+ 342us) server_negotiation.cc:407] Connection header received
0807 06:48:21.949849 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:21.949858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:21.949989 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:21.950137 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:21.951790 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.952526 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.953995 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:21.954455 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:21.954596 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:21.959143 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:21.959149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:21.959152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:21.959192 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:21.962075 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.962961 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.962966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.962969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.963123 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:21.963513 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:21.963517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:21.963519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:21.963815 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:21.964048 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:21.964301 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 06:48:21.964511 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 06:48:21.969512 29439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:21.976858 29439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:22.012395 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.997962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.27.54.67:39983)
0807 06:48:22.004899 (+ 6937us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.004910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.004946 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:48:22.005039 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.005046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.005265 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.005476 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.006098 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.007669 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.009715 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.011091 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.011231 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.011502 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.011797 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.011974 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6754,"thread_start_us":6558,"threads_started":1}
I20260807 06:48:22.012503 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.997863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:39983 (local address 127.0.0.1:34338)
0807 06:48:22.004102 (+ 6239us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.004125 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.004242 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.005477 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.005481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.005505 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.005886 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.005898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.007805 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.007819 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.009542 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.009556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.011287 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.011294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.011520 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.011573 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.011801 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.012005 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6109,"mutex_wait_us":21,"thread_start_us":5978,"threads_started":1}
I20260807 06:48:22.013705 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.989475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44445 (local address 127.0.0.1:50116)
0807 06:48:21.996245 (+ 6770us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:21.996315 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:21.996489 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.004544 (+ 8055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.004555 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.004591 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.005268 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.005289 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.007325 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.007331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.010304 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.010320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.012805 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.012817 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.013022 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.013057 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.013162 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.013304 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6512,"mutex_wait_us":191,"thread_start_us":7209,"threads_started":1}
I20260807 06:48:22.014091 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.990199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:46203 (local address 127.0.0.1:60630)
0807 06:48:22.002952 (+ 12753us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.002989 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.003166 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.004822 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.004827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.004855 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.005341 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.005351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.007325 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.007331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.010304 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.010320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.012230 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.012236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.012429 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.013265 (+ 836us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.013370 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.013513 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12553,"mutex_wait_us":221,"thread_start_us":11109,"threads_started":1}
I20260807 06:48:22.014796 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.990902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.27.54.66:46203)
0807 06:48:21.991685 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:21.991699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.004113 (+ 12414us) server_negotiation.cc:407] Connection header received
0807 06:48:22.004277 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.004288 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.004572 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.004844 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.005533 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.007153 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.010483 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.011967 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.012779 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.012989 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.014352 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.014498 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":355,"threads_started":1}
I20260807 06:48:22.015640 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:21.989870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.27.54.65:44445)
0807 06:48:22.003752 (+ 13882us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.003764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.003801 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:48:22.003935 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.003945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.004259 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.004981 (+ 722us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.005524 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.007157 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.010512 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.012017 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.012164 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.012409 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.015155 (+ 2746us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.015342 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13710,"thread_start_us":13577,"threads_started":1}
I20260807 06:48:22.016535 30049 tablet_service.cc:1467] Processing CreateTablet for tablet cd5a474d6ae64d8f846e724ccf645ee5 (DEFAULT_TABLE table=test-table [id=d35b6096638a4820bc7d95ec6d289744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:22.017830 30049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5a474d6ae64d8f846e724ccf645ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.017750 29845 tablet_service.cc:1467] Processing CreateTablet for tablet cd5a474d6ae64d8f846e724ccf645ee5 (DEFAULT_TABLE table=test-table [id=d35b6096638a4820bc7d95ec6d289744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:22.019054 29845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5a474d6ae64d8f846e724ccf645ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.022260 29643 tablet_service.cc:1467] Processing CreateTablet for tablet cd5a474d6ae64d8f846e724ccf645ee5 (DEFAULT_TABLE table=test-table [id=d35b6096638a4820bc7d95ec6d289744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:22.023726 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd5a474d6ae64d8f846e724ccf645ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.031642 30178 tablet_bootstrap.cc:492] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Bootstrap starting.
I20260807 06:48:22.033887 30181 tablet_bootstrap.cc:492] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Bootstrap starting.
I20260807 06:48:22.035872 30178 tablet_bootstrap.cc:654] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.037361 30178 log.cc:824] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:22.038021 30181 tablet_bootstrap.cc:654] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.038424 30182 tablet_bootstrap.cc:492] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Bootstrap starting.
I20260807 06:48:22.039789 30181 log.cc:824] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:22.042168 30182 tablet_bootstrap.cc:654] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.043483 30182 log.cc:824] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:22.046938 30182 tablet_bootstrap.cc:492] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: No bootstrap required, opened a new log
I20260807 06:48:22.047137 30178 tablet_bootstrap.cc:492] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: No bootstrap required, opened a new log
I20260807 06:48:22.047199 30181 tablet_bootstrap.cc:492] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: No bootstrap required, opened a new log
I20260807 06:48:22.047266 30182 ts_tablet_manager.cc:1397] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260807 06:48:22.047456 30178 ts_tablet_manager.cc:1397] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260807 06:48:22.047509 30181 ts_tablet_manager.cc:1397] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260807 06:48:22.055284 30182 raft_consensus.cc:348] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.055732 30182 raft_consensus.cc:374] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.055837 30182 raft_consensus.cc:729] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20d5944df6649539e2c25443a74e9b3, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.056528 30182 consensus_queue.cc:260] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.058568 30182 ts_tablet_manager.cc:1428] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260807 06:48:22.058725 30178 raft_consensus.cc:348] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.058686 30181 raft_consensus.cc:348] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.059173 30178 raft_consensus.cc:374] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.059173 30181 raft_consensus.cc:374] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.059288 30181 raft_consensus.cc:729] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ca5e28384643bf810e478b54e51afd, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.059288 30178 raft_consensus.cc:729] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a16cf2cbcb545e8bb5b0081ce33e61e, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.059966 30178 consensus_queue.cc:260] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.059979 30181 consensus_queue.cc:260] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.061281 30151 heartbeater.cc:503] Master 127.27.54.126:39259 was elected leader, sending a full tablet report...
I20260807 06:48:22.062089 29947 heartbeater.cc:503] Master 127.27.54.126:39259 was elected leader, sending a full tablet report...
I20260807 06:48:22.062891 30178 ts_tablet_manager.cc:1428] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260807 06:48:22.064457 30181 ts_tablet_manager.cc:1428] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
W20260807 06:48:22.065387 29949 tablet.cc:2369] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:22.110605 30188 raft_consensus.cc:482] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:22.111070 30188 raft_consensus.cc:504] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.112851 30188 leader_election.cc:288] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445), 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203)
I20260807 06:48:22.119122 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.113284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:55029 (local address 127.27.54.65:44445)
0807 06:48:22.113502 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.113508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.113532 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:48:22.113738 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.113746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.113833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.113920 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.114608 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.115689 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.117348 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.118177 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.118350 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.118425 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.118714 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.118849 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:48:22.119184 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.113103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44445 (local address 127.27.54.67:55029)
0807 06:48:22.113359 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.113382 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.113527 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.114126 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.114130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.114177 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.114423 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.114439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.115812 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.115818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.117188 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.117201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.118344 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.118350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.118527 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.118551 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.118684 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.118825 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":2}
I20260807 06:48:22.119791 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.113268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:46203 (local address 127.27.54.67:55493)
0807 06:48:22.114358 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.114381 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.114509 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.114875 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.114879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.114907 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.115251 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.115263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.116321 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.116327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.117716 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.117730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.118948 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.118954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.119114 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.119136 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.119256 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.119411 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":1,"thread_start_us":705,"threads_started":1}
I20260807 06:48:22.120153 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.113687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:55493 (local address 127.27.54.66:46203)
0807 06:48:22.113923 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.113928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.114475 (+ 547us) server_negotiation.cc:407] Connection header received
0807 06:48:22.114624 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.114629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.114740 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.114844 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.115441 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.116173 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.117886 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.118723 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.118904 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.118983 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.119697 (+ 714us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.119875 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:48:22.120332 29682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5a474d6ae64d8f846e724ccf645ee5" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20d5944df6649539e2c25443a74e9b3" is_pre_election: true
I20260807 06:48:22.120848 29682 raft_consensus.cc:2454] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 0.
I20260807 06:48:22.121022 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5a474d6ae64d8f846e724ccf645ee5" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" is_pre_election: true
I20260807 06:48:22.121547 29879 raft_consensus.cc:2454] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 0.
I20260807 06:48:22.121531 29979 leader_election.cc:302] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ca5e28384643bf810e478b54e51afd, e20d5944df6649539e2c25443a74e9b3; no voters:
I20260807 06:48:22.122022 30188 raft_consensus.cc:2781] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:22.122233 30188 raft_consensus.cc:482] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:22.122361 30188 raft_consensus.cc:3037] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.125032 30188 raft_consensus.cc:504] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.125779 30188 leader_election.cc:288] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 election: Requested vote from peers e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445), 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203)
I20260807 06:48:22.126335 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5a474d6ae64d8f846e724ccf645ee5" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e"
I20260807 06:48:22.126341 29682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd5a474d6ae64d8f846e724ccf645ee5" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20d5944df6649539e2c25443a74e9b3"
I20260807 06:48:22.126556 29879 raft_consensus.cc:3037] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.126569 29682 raft_consensus.cc:3037] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.128973 29682 raft_consensus.cc:2454] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 1.
I20260807 06:48:22.128979 29879 raft_consensus.cc:2454] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 1.
I20260807 06:48:22.129498 29979 leader_election.cc:302] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ca5e28384643bf810e478b54e51afd, e20d5944df6649539e2c25443a74e9b3; no voters:
I20260807 06:48:22.129882 30188 raft_consensus.cc:2781] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:22.130431 30188 raft_consensus.cc:686] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 1 LEADER]: Becoming Leader. State: Replica: 55ca5e28384643bf810e478b54e51afd, State: Running, Role: LEADER
I20260807 06:48:22.131042 30188 consensus_queue.cc:237] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } }
I20260807 06:48:22.136230 29438 catalog_manager.cc:5495] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd reported cstate change: term changed from 0 to 1, leader changed from <none> to 55ca5e28384643bf810e478b54e51afd (127.27.54.67). New cstate: current_term: 1 leader_uuid: "55ca5e28384643bf810e478b54e51afd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } health_report { overall_health: UNKNOWN } } }
W20260807 06:48:22.140882 30152 tablet.cc:2369] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:22.170109 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.164305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.27.54.67:39983)
0807 06:48:22.164552 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.164557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.164577 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:48:22.164943 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.164947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.165039 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.165181 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:22.165539 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.166310 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.167728 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.168205 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.168318 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.168836 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.169466 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.169591 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.169697 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.169790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:48:22.185240 29682 raft_consensus.cc:1261] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Refusing update from remote peer 55ca5e28384643bf810e478b54e51afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:22.185240 29879 raft_consensus.cc:1261] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Refusing update from remote peer 55ca5e28384643bf810e478b54e51afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:22.186478 30188 consensus_queue.cc:1035] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:22.188530 30188 consensus_queue.cc:1035] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:22.224572 29438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:22.225001 29438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:22.229269 29438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: Sending DeleteTablet for 3 replicas of tablet cd5a474d6ae64d8f846e724ccf645ee5
I20260807 06:48:22.230882 29643 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5a474d6ae64d8f846e724ccf645ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:22 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:50116
I20260807 06:48:22.231074 29845 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5a474d6ae64d8f846e724ccf645ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:22 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60630
I20260807 06:48:22.231931 30049 tablet_service.cc:1514] Processing DeleteTablet for tablet cd5a474d6ae64d8f846e724ccf645ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:22 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:34338
I20260807 06:48:22.233382 29438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:22.234359 29438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:22.236552 30209 tablet_replica.cc:330] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: stopping tablet replica
I20260807 06:48:22.237233 30209 raft_consensus.cc:2229] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:22.238124 30209 raft_consensus.cc:2258] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:22.239053 30208 tablet_replica.cc:330] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: stopping tablet replica
I20260807 06:48:22.239539 30208 raft_consensus.cc:2229] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:22.240087 30208 raft_consensus.cc:2258] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:22.241611 30207 tablet_replica.cc:330] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: stopping tablet replica
I20260807 06:48:22.242050 30207 raft_consensus.cc:2229] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:22.242579 30207 raft_consensus.cc:2258] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:22.242898 30209 ts_tablet_manager.cc:1905] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:22.244402 30208 ts_tablet_manager.cc:1905] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:22.244982 29845 tablet_service.cc:1467] Processing CreateTablet for tablet a3183ef9719b4437ac05ab63aad0b69e (DEFAULT_TABLE table=test-table [id=c9c75b2db7694db2975213629bfd81f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:22.245457 29845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3183ef9719b4437ac05ab63aad0b69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.251685 29643 tablet_service.cc:1467] Processing CreateTablet for tablet a3183ef9719b4437ac05ab63aad0b69e (DEFAULT_TABLE table=test-table [id=c9c75b2db7694db2975213629bfd81f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:22.252238 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3183ef9719b4437ac05ab63aad0b69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.253135 30178 tablet_bootstrap.cc:492] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Bootstrap starting.
I20260807 06:48:22.254510 30207 ts_tablet_manager.cc:1905] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:22.255048 30178 tablet_bootstrap.cc:654] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.258096 30178 tablet_bootstrap.cc:492] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: No bootstrap required, opened a new log
I20260807 06:48:22.258373 30178 ts_tablet_manager.cc:1397] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260807 06:48:22.259202 30209 ts_tablet_manager.cc:1918] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:22.259524 30209 log.cc:1196] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/cd5a474d6ae64d8f846e724ccf645ee5
I20260807 06:48:22.259689 30208 ts_tablet_manager.cc:1918] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:22.259909 30208 log.cc:1196] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/cd5a474d6ae64d8f846e724ccf645ee5
I20260807 06:48:22.259876 30178 raft_consensus.cc:348] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.260512 30208 ts_tablet_manager.cc:1939] T cd5a474d6ae64d8f846e724ccf645ee5 P e20d5944df6649539e2c25443a74e9b3: Deleting consensus metadata
I20260807 06:48:22.260704 30209 ts_tablet_manager.cc:1939] T cd5a474d6ae64d8f846e724ccf645ee5 P 55ca5e28384643bf810e478b54e51afd: Deleting consensus metadata
I20260807 06:48:22.260914 30049 tablet_service.cc:1467] Processing CreateTablet for tablet a3183ef9719b4437ac05ab63aad0b69e (DEFAULT_TABLE table=test-table [id=c9c75b2db7694db2975213629bfd81f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:22.261529 30049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3183ef9719b4437ac05ab63aad0b69e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.260263 30178 raft_consensus.cc:374] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.262117 30178 raft_consensus.cc:729] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a16cf2cbcb545e8bb5b0081ce33e61e, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.262575 30178 consensus_queue.cc:260] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.262751 29419 catalog_manager.cc:4841] TS e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445): tablet cd5a474d6ae64d8f846e724ccf645ee5 (table test-table [id=d35b6096638a4820bc7d95ec6d289744]) successfully deleted
I20260807 06:48:22.263391 30178 ts_tablet_manager.cc:1428] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260807 06:48:22.263517 30182 tablet_bootstrap.cc:492] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Bootstrap starting.
I20260807 06:48:22.262178 30207 ts_tablet_manager.cc:1918] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:22.263873 30207 log.cc:1196] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/cd5a474d6ae64d8f846e724ccf645ee5
I20260807 06:48:22.264556 30207 ts_tablet_manager.cc:1939] T cd5a474d6ae64d8f846e724ccf645ee5 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Deleting consensus metadata
I20260807 06:48:22.265381 30182 tablet_bootstrap.cc:654] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.267156 29413 catalog_manager.cc:4841] TS 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203): tablet cd5a474d6ae64d8f846e724ccf645ee5 (table test-table [id=d35b6096638a4820bc7d95ec6d289744]) successfully deleted
I20260807 06:48:22.269213 29419 catalog_manager.cc:4841] TS 55ca5e28384643bf810e478b54e51afd (127.27.54.67:39983): tablet cd5a474d6ae64d8f846e724ccf645ee5 (table test-table [id=d35b6096638a4820bc7d95ec6d289744]) successfully deleted
I20260807 06:48:22.272369 30181 tablet_bootstrap.cc:492] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Bootstrap starting.
I20260807 06:48:22.273794 30182 tablet_bootstrap.cc:492] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: No bootstrap required, opened a new log
I20260807 06:48:22.273964 30182 ts_tablet_manager.cc:1397] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 06:48:22.274250 30181 tablet_bootstrap.cc:654] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.274930 30182 raft_consensus.cc:348] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.275180 30182 raft_consensus.cc:374] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.275290 30182 raft_consensus.cc:729] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20d5944df6649539e2c25443a74e9b3, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.275627 30182 consensus_queue.cc:260] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.276362 30182 ts_tablet_manager.cc:1428] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 06:48:22.277052 30181 tablet_bootstrap.cc:492] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: No bootstrap required, opened a new log
I20260807 06:48:22.277245 30181 ts_tablet_manager.cc:1397] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:48:22.278165 30181 raft_consensus.cc:348] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.278419 30181 raft_consensus.cc:374] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.278532 30181 raft_consensus.cc:729] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ca5e28384643bf810e478b54e51afd, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.278872 30181 consensus_queue.cc:260] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.279515 30181 ts_tablet_manager.cc:1428] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 06:48:22.371856 30201 raft_consensus.cc:482] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:22.372153 30201 raft_consensus.cc:504] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.373024 30201 leader_election.cc:288] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445), 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203)
I20260807 06:48:22.373726 29682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3183ef9719b4437ac05ab63aad0b69e" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20d5944df6649539e2c25443a74e9b3" is_pre_election: true
I20260807 06:48:22.373721 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3183ef9719b4437ac05ab63aad0b69e" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" is_pre_election: true
I20260807 06:48:22.374020 29682 raft_consensus.cc:2454] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 0.
I20260807 06:48:22.374022 29879 raft_consensus.cc:2454] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 0.
I20260807 06:48:22.374619 29976 leader_election.cc:302] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a16cf2cbcb545e8bb5b0081ce33e61e, 55ca5e28384643bf810e478b54e51afd, e20d5944df6649539e2c25443a74e9b3; no voters:
I20260807 06:48:22.375007 30201 raft_consensus.cc:2781] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:22.375149 30201 raft_consensus.cc:482] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:22.375245 30201 raft_consensus.cc:3037] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.377734 30201 raft_consensus.cc:504] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.378383 30201 leader_election.cc:288] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 election: Requested vote from peers e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445), 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203)
I20260807 06:48:22.379004 29682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3183ef9719b4437ac05ab63aad0b69e" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20d5944df6649539e2c25443a74e9b3"
I20260807 06:48:22.379047 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3183ef9719b4437ac05ab63aad0b69e" candidate_uuid: "55ca5e28384643bf810e478b54e51afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e"
I20260807 06:48:22.379269 29682 raft_consensus.cc:3037] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.379269 29879 raft_consensus.cc:3037] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.382418 29682 raft_consensus.cc:2454] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 1.
I20260807 06:48:22.382418 29879 raft_consensus.cc:2454] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ca5e28384643bf810e478b54e51afd in term 1.
I20260807 06:48:22.382989 29976 leader_election.cc:302] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a16cf2cbcb545e8bb5b0081ce33e61e, 55ca5e28384643bf810e478b54e51afd; no voters:
I20260807 06:48:22.383363 30201 raft_consensus.cc:2781] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:22.383560 30201 raft_consensus.cc:686] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 1 LEADER]: Becoming Leader. State: Replica: 55ca5e28384643bf810e478b54e51afd, State: Running, Role: LEADER
I20260807 06:48:22.383869 30201 consensus_queue.cc:237] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.387053 29437 catalog_manager.cc:5495] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd reported cstate change: term changed from 0 to 1, leader changed from <none> to 55ca5e28384643bf810e478b54e51afd (127.27.54.67). New cstate: current_term: 1 leader_uuid: "55ca5e28384643bf810e478b54e51afd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } health_report { overall_health: HEALTHY } } }
Aug 07 06:48:22 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:22.446449 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.429761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39259 (local address 127.0.0.1:34880)
0807 06:48:22.430521 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.430539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.430759 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.431160 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.431163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.431962 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:22.432237 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.432247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.433622 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.433626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.434827 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.434844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.435994 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.435999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.436121 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.436986 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:22.437004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:22.439385 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:22.443481 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:22.443495 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:22.443502 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:22.443983 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:22.444465 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:22.444469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:22.444473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:22.444665 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:22.445342 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:22.445348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:22.445560 (+ 212us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.445760 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.446095 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":402,"threads_started":1}
I20260807 06:48:22.446722 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.429672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.27.54.126:39259)
0807 06:48:22.429923 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.429931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.430669 (+ 738us) server_negotiation.cc:407] Connection header received
0807 06:48:22.430890 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.430897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.430978 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.431107 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:22.432418 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.433473 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.434992 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.435748 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.435890 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.439610 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:22.439617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:22.439621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:22.439660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:22.443308 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.444128 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.444134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.444138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.444310 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.444793 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.444799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.444803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.445199 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:22.445363 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.445913 (+ 550us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.446292 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":86}
I20260807 06:48:22.452594 29437 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:34880:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 06:48:22.453985 29437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:22.461575 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa4f294beff41f7a8447f9af3370e00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cbf5800000548d59578db37f675babb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:48:22.461843 29845 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa4f294beff41f7a8447f9af3370e00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cbf5800000548d59578db37f675babb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:48:22.461949 30049 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa4f294beff41f7a8447f9af3370e00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cbf5800000548d59578db37f675babb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:48:22.462267 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa4f294beff41f7a8447f9af3370e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.462509 30049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa4f294beff41f7a8447f9af3370e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.462502 29845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa4f294beff41f7a8447f9af3370e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:22.472085 30178 tablet_bootstrap.cc:492] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Bootstrap starting.
I20260807 06:48:22.472368 30182 tablet_bootstrap.cc:492] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: Bootstrap starting.
I20260807 06:48:22.474187 30182 tablet_bootstrap.cc:654] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.474187 30178 tablet_bootstrap.cc:654] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.477180 30181 tablet_bootstrap.cc:492] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: Bootstrap starting.
I20260807 06:48:22.479310 30181 tablet_bootstrap.cc:654] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:22.483516 30181 tablet_bootstrap.cc:492] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: No bootstrap required, opened a new log
I20260807 06:48:22.483810 30181 ts_tablet_manager.cc:1397] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260807 06:48:22.485711 30181 raft_consensus.cc:348] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.486109 30181 raft_consensus.cc:374] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.486423 30181 raft_consensus.cc:729] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ca5e28384643bf810e478b54e51afd, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.486927 30181 consensus_queue.cc:260] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.487620 30201 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.487933 30201 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.489277 30181 ts_tablet_manager.cc:1428] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 06:48:22.489329 30182 tablet_bootstrap.cc:492] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: No bootstrap required, opened a new log
I20260807 06:48:22.489657 30182 ts_tablet_manager.cc:1397] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260807 06:48:22.490590 30182 raft_consensus.cc:348] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.490907 30182 raft_consensus.cc:374] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.491001 30182 raft_consensus.cc:729] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20d5944df6649539e2c25443a74e9b3, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.491376 30182 consensus_queue.cc:260] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.491933 30186 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.492303 30186 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.492292 30178 tablet_bootstrap.cc:492] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: No bootstrap required, opened a new log
I20260807 06:48:22.492487 30178 ts_tablet_manager.cc:1397] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.011s
W20260807 06:48:22.493235 29748 tablet.cc:2369] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:22.493283 30182 ts_tablet_manager.cc:1428] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:48:22.493346 30178 raft_consensus.cc:348] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.493615 30178 raft_consensus.cc:374] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:22.493675 30178 raft_consensus.cc:729] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a16cf2cbcb545e8bb5b0081ce33e61e, State: Initialized, Role: FOLLOWER
I20260807 06:48:22.493963 30178 consensus_queue.cc:260] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.494586 30178 ts_tablet_manager.cc:1428] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 06:48:22.494438 30187 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.494872 30187 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.604171 30186 raft_consensus.cc:482] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:22.604461 30186 raft_consensus.cc:504] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.606671 30186 leader_election.cc:288] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203), 55ca5e28384643bf810e478b54e51afd (127.27.54.67:39983)
I20260807 06:48:22.615965 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.607069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:38991 (local address 127.27.54.66:46203)
0807 06:48:22.607306 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.607313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.608830 (+ 1517us) server_negotiation.cc:407] Connection header received
0807 06:48:22.609055 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.609061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.609132 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.609184 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.610047 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.611298 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.613838 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.614752 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.614959 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.615043 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.615471 (+ 428us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.615627 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 06:48:22.616071 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.607007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:46203 (local address 127.27.54.65:38991)
0807 06:48:22.608698 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.608717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.608862 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.609376 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.609379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.609398 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.609852 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.609865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.611440 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.611446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.613679 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.613695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.614996 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.615003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.615201 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.615251 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.615534 (+ 283us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.615699 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1576,"thread_start_us":858,"threads_started":1}
I20260807 06:48:22.616137 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.607482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:59397 (local address 127.27.54.67:39983)
0807 06:48:22.607663 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.607670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.608344 (+ 674us) server_negotiation.cc:407] Connection header received
0807 06:48:22.608661 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.608667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.608787 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.608898 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:22.609611 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.610804 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.613661 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.615058 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.615253 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.615342 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.615686 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.615846 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:48:22.616899 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.606944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:39983 (local address 127.27.54.65:59397)
0807 06:48:22.608204 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.608234 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.608488 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.608937 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.608944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.608977 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:22.609375 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.609391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.610961 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.610966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.613445 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.613463 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.615279 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.615284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.615493 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.615526 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.616334 (+ 808us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.616494 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":1,"thread_start_us":1422,"threads_started":1}
I20260807 06:48:22.617268 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa4f294beff41f7a8447f9af3370e00" candidate_uuid: "e20d5944df6649539e2c25443a74e9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" is_pre_election: true
I20260807 06:48:22.617619 29879 raft_consensus.cc:2454] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e20d5944df6649539e2c25443a74e9b3 in term 0.
I20260807 06:48:22.617970 30083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa4f294beff41f7a8447f9af3370e00" candidate_uuid: "e20d5944df6649539e2c25443a74e9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ca5e28384643bf810e478b54e51afd" is_pre_election: true
I20260807 06:48:22.618193 29571 leader_election.cc:302] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a16cf2cbcb545e8bb5b0081ce33e61e, e20d5944df6649539e2c25443a74e9b3; no voters:
I20260807 06:48:22.618459 30083 raft_consensus.cc:2454] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e20d5944df6649539e2c25443a74e9b3 in term 0.
I20260807 06:48:22.618716 30186 raft_consensus.cc:2781] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:22.618858 30186 raft_consensus.cc:482] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:22.618985 30186 raft_consensus.cc:3037] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.621680 30186 raft_consensus.cc:504] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.622288 30186 leader_election.cc:288] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [CANDIDATE]: Term 1 election: Requested vote from peers 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203), 55ca5e28384643bf810e478b54e51afd (127.27.54.67:39983)
I20260807 06:48:22.622964 30083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa4f294beff41f7a8447f9af3370e00" candidate_uuid: "e20d5944df6649539e2c25443a74e9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ca5e28384643bf810e478b54e51afd"
I20260807 06:48:22.623208 30083 raft_consensus.cc:3037] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.626093 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa4f294beff41f7a8447f9af3370e00" candidate_uuid: "e20d5944df6649539e2c25443a74e9b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e"
I20260807 06:48:22.626397 29879 raft_consensus.cc:3037] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:22.627573 30083 raft_consensus.cc:2454] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e20d5944df6649539e2c25443a74e9b3 in term 1.
I20260807 06:48:22.628121 29576 leader_election.cc:302] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ca5e28384643bf810e478b54e51afd, e20d5944df6649539e2c25443a74e9b3; no voters:
I20260807 06:48:22.628492 30186 raft_consensus.cc:2781] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:22.629583 30186 raft_consensus.cc:686] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [term 1 LEADER]: Becoming Leader. State: Replica: e20d5944df6649539e2c25443a74e9b3, State: Running, Role: LEADER
I20260807 06:48:22.629730 29879 raft_consensus.cc:2454] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e20d5944df6649539e2c25443a74e9b3 in term 1.
I20260807 06:48:22.630093 30186 consensus_queue.cc:237] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } }
I20260807 06:48:22.632556 30234 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: TxnStatusTablet state changed. Reason: New leader e20d5944df6649539e2c25443a74e9b3. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.632931 30234 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:48:22.633798 30239 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 06:48:22.634181 30239 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 06:48:22.634786 29437 catalog_manager.cc:5495] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e20d5944df6649539e2c25443a74e9b3 (127.27.54.65). New cstate: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } health_report { overall_health: UNKNOWN } } }
I20260807 06:48:22.696038 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.682798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.27.54.65:44445)
0807 06:48:22.688132 (+ 5334us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.688139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.688174 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:48:22.688327 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.688334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.688464 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.688784 (+ 320us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:22.689305 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.690703 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.692677 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.693424 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.693571 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.694243 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.695184 (+ 941us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.695373 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.695517 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.695629 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5156,"thread_start_us":4961,"threads_started":1}
I20260807 06:48:22.695995 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.683057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44445 (local address 127.0.0.1:50122)
0807 06:48:22.683349 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.683374 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.683537 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.688838 (+ 5301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.688844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.688860 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:22.689094 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.689106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.690864 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.690871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.692482 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.692502 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.693627 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.693634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.693881 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.693975 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.695307 (+ 1332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.695337 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.695448 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.695627 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 06:48:22.702198 29879 raft_consensus.cc:1261] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Refusing update from remote peer e20d5944df6649539e2c25443a74e9b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:22.702346 30083 raft_consensus.cc:1261] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Refusing update from remote peer e20d5944df6649539e2c25443a74e9b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:22.703361 30234 consensus_queue.cc:1035] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:22.703807 30186 consensus_queue.cc:1035] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:22.726121 30201 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: TxnStatusTablet state changed. Reason: New leader e20d5944df6649539e2c25443a74e9b3. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.726505 30201 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.727705 30187 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: TxnStatusTablet state changed. Reason: New leader e20d5944df6649539e2c25443a74e9b3. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.727855 30195 mvcc.cc:205] Tried to move back new op lower bound from 7315805399858634752 to 7315805399577120768. Current Snapshot: MvccSnapshot[applied={T|T < 7315805399858634752}]
I20260807 06:48:22.728027 30187 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.730419 30234 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.730623 30234 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:48:22.733901 30201 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.734212 30201 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P 55ca5e28384643bf810e478b54e51afd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.737229 30186 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.737537 30186 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P e20d5944df6649539e2c25443a74e9b3: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:48:22.759025 30187 tablet_replica.cc:439] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e20d5944df6649539e2c25443a74e9b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 } } peers { permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 } } peers { permanent_uuid: "55ca5e28384643bf810e478b54e51afd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39983 } } }
I20260807 06:48:22.759337 30187 tablet_replica.cc:442] T 3aa4f294beff41f7a8447f9af3370e00 P 3a16cf2cbcb545e8bb5b0081ce33e61e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:22.768138 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.759786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:55241 (local address 127.27.54.65:44445)
0807 06:48:22.760025 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.760031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.760834 (+ 803us) server_negotiation.cc:407] Connection header received
0807 06:48:22.761205 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.761212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.761386 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.761497 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:22.762297 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.763586 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.765746 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.766593 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.766771 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.767276 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.767427 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.767534 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.767701 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.767796 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 06:48:22.768215 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.760008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44445 (local address 127.27.54.67:55241)
0807 06:48:22.760721 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.760743 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.761028 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.761717 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.761722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.761735 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:22.762062 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.762073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.763741 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.763747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.765491 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.765512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.766820 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.766826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.767003 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.767081 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.767549 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.767571 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.767659 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.767821 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":370,"threads_started":1}
I20260807 06:48:22.781051 29879 raft_consensus.cc:1261] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Refusing update from remote peer 55ca5e28384643bf810e478b54e51afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:22.781951 30188 consensus_queue.cc:1035] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e20d5944df6649539e2c25443a74e9b3" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:22.782311 30201 consensus_queue.cc:1035] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a16cf2cbcb545e8bb5b0081ce33e61e" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:22.790431 30247 mvcc.cc:205] Tried to move back new op lower bound from 7315805400185921536 to 7315805398567112704. Current Snapshot: MvccSnapshot[applied={T|T < 7315805400185921536}]
I20260807 06:48:22.827782 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.820111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:41975 (local address 127.27.54.67:39983)
0807 06:48:22.820318 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.820324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.820791 (+ 467us) server_negotiation.cc:407] Connection header received
0807 06:48:22.821049 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.821056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.821147 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.821239 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:22.822154 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.823043 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.825503 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.826216 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.826363 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.827014 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.827147 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.827249 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.827372 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.827461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:48:22.827978 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.819933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:39983 (local address 127.27.54.65:41975)
0807 06:48:22.820685 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:22.820703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:22.820868 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:22.821416 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:22.821421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:22.821435 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:22.821969 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.821982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.823175 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.823182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.825304 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:22.825321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.826440 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.826447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:22.826684 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.826793 (+ 109us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:22.827248 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:22.827260 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:48:22.827348 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 06:48:22.827505 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":393,"threads_started":1}
I20260807 06:48:22.874748 29437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:22.875088 29437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:22.878244 29437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9bab06eaeab4ba88d7f59a279aa5df8: Sending DeleteTablet for 3 replicas of tablet a3183ef9719b4437ac05ab63aad0b69e
I20260807 06:48:22.879237 29845 tablet_service.cc:1514] Processing DeleteTablet for tablet a3183ef9719b4437ac05ab63aad0b69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:22 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60630
I20260807 06:48:22.879242 29643 tablet_service.cc:1514] Processing DeleteTablet for tablet a3183ef9719b4437ac05ab63aad0b69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:22 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:50116
I20260807 06:48:22.879633 30049 tablet_service.cc:1514] Processing DeleteTablet for tablet a3183ef9719b4437ac05ab63aad0b69e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:22 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:34338
I20260807 06:48:22.889380 30274 tablet_replica.cc:330] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: stopping tablet replica
I20260807 06:48:22.889698 30274 raft_consensus.cc:2229] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:22.890141 30274 raft_consensus.cc:2258] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:22.892004 30274 ts_tablet_manager.cc:1905] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:22.902241 30274 ts_tablet_manager.cc:1918] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:22.902415 30274 log.cc:1196] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/a3183ef9719b4437ac05ab63aad0b69e
I20260807 06:48:22.903060 30274 ts_tablet_manager.cc:1939] T a3183ef9719b4437ac05ab63aad0b69e P e20d5944df6649539e2c25443a74e9b3: Deleting consensus metadata
I20260807 06:48:22.904323 29419 catalog_manager.cc:4841] TS e20d5944df6649539e2c25443a74e9b3 (127.27.54.65:44445): tablet a3183ef9719b4437ac05ab63aad0b69e (table test-table [id=c9c75b2db7694db2975213629bfd81f6]) successfully deleted
I20260807 06:48:22.905544 30279 tablet_replica.cc:330] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: stopping tablet replica
I20260807 06:48:22.905781 30279 raft_consensus.cc:2229] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:22.906248 30279 raft_consensus.cc:2258] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:22.907109 30272 tablet_replica.cc:330] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: stopping tablet replica
I20260807 06:48:22.907464 30272 raft_consensus.cc:2229] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:22.907742 30272 raft_consensus.cc:2258] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:22.908910 30272 ts_tablet_manager.cc:1905] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:22.910601 30279 ts_tablet_manager.cc:1905] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:22.916282 30272 ts_tablet_manager.cc:1918] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:22.916484 30272 log.cc:1196] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/a3183ef9719b4437ac05ab63aad0b69e
I20260807 06:48:22.917028 30272 ts_tablet_manager.cc:1939] T a3183ef9719b4437ac05ab63aad0b69e P 3a16cf2cbcb545e8bb5b0081ce33e61e: Deleting consensus metadata
I20260807 06:48:22.918231 29413 catalog_manager.cc:4841] TS 3a16cf2cbcb545e8bb5b0081ce33e61e (127.27.54.66:46203): tablet a3183ef9719b4437ac05ab63aad0b69e (table test-table [id=c9c75b2db7694db2975213629bfd81f6]) successfully deleted
I20260807 06:48:22.921864 30279 ts_tablet_manager.cc:1918] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:22.922039 30279 log.cc:1196] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/a3183ef9719b4437ac05ab63aad0b69e
I20260807 06:48:22.922544 30279 ts_tablet_manager.cc:1939] T a3183ef9719b4437ac05ab63aad0b69e P 55ca5e28384643bf810e478b54e51afd: Deleting consensus metadata
Aug 07 06:48:22 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:22.923842 29419 catalog_manager.cc:4841] TS 55ca5e28384643bf810e478b54e51afd (127.27.54.67:39983): tablet a3183ef9719b4437ac05ab63aad0b69e (table test-table [id=c9c75b2db7694db2975213629bfd81f6]) successfully deleted
I20260807 06:48:22.932883 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.902694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.27.54.65:44445)
0807 06:48:22.902923 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.902932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.907079 (+ 4147us) server_negotiation.cc:407] Connection header received
0807 06:48:22.907378 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.907384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.907491 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.907575 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:22.912554 (+ 4979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.913814 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.918096 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.918872 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.918976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.924919 (+ 5943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:22.924933 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:22.924954 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:22.925008 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:22.929334 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.930225 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.930235 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.930240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.930402 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.930855 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.930861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.930864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.931355 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:22.931740 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.932010 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.932250 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260807 06:48:22.933667 29737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50124
I20260807 06:48:22.953171 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.937777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.27.54.126:39259)
0807 06:48:22.937958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.937962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.939236 (+ 1274us) server_negotiation.cc:407] Connection header received
0807 06:48:22.939391 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.939397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.939480 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.939582 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:22.941410 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.942159 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.943938 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.944396 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.944509 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.947586 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:22.947591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:22.947594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:22.947632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:22.950395 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.951234 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.951238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.951240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.951384 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.951808 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.951812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.951815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.952092 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:22.952266 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.952703 (+ 437us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.952895 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 06:48:22.953847 29437 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34882
I20260807 06:48:22.956861 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29544
I20260807 06:48:22.973022 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:22.957918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.27.54.65:44445)
0807 06:48:22.958179 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:22.958185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:22.958366 (+ 181us) server_negotiation.cc:407] Connection header received
0807 06:48:22.958553 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:22.958557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:22.958623 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:22.958756 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:22.960480 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.961330 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.963332 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:22.963887 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:22.963992 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:22.967058 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:22.967063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:22.967066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:22.967103 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:22.970129 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.971025 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.971029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.971032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.971168 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:22.971567 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:22.971570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:22.971572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:22.971876 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:22.972083 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:22.972470 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 06:48:22.972662 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260807 06:48:22.973815 29737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50130
W20260807 06:48:22.974411 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:22.974630 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 29544
I20260807 06:48:23.008046 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29751
Aug 07 06:48:23 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:23.023988 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:23.008866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.27.54.66:46203)
0807 06:48:23.009036 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:23.009041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:23.009126 (+ 85us) server_negotiation.cc:407] Connection header received
0807 06:48:23.009337 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:23.009342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:23.009416 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:23.009504 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:23.010760 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:23.011507 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:23.012957 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:23.013410 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:23.013555 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:23.017910 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:23.017925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:23.017941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:23.017977 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:23.020973 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:23.021674 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:23.021682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:23.021684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:23.021821 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:23.022211 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:23.022216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:23.022219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:23.022574 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:23.023069 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:23.023361 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 06:48:23.023604 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260807 06:48:23.024715 29938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632
W20260807 06:48:23.025480 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29751: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:23.025650 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 29751
I20260807 06:48:23.051779 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29953
Aug 07 06:48:23 dist-test-slave-3kk6 krb5kdc[29369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:23.068213 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:23.052631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.27.54.67:39983)
0807 06:48:23.052843 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:23.052851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:23.052875 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:48:23.053097 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:23.053101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:23.053166 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:23.053289 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:23.054626 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:23.055421 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:23.056811 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:23.057239 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:23.057392 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:23.061955 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:23.061973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:23.061990 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:23.062027 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:23.065003 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:23.065732 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:23.065740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:23.065743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:23.065890 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:23.066276 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:23.066284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:23.066287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:23.066888 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:23.067262 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:23.067601 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 06:48:23.067872 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 06:48:23.068925 30142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34350
W20260807 06:48:23.069576 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29953: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:23.069785 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 29953
I20260807 06:48:23.107666 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 29389
I20260807 06:48:23.108989 29499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:23.230026 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 29389
2026-08-07T06:48:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5848 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:23 dist-test-slave-3kk6 krb5kdc[30293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:23 dist-test-slave-3kk6 krb5kdc[30293](info): set up 2 sockets
Aug 07 06:48:23 dist-test-slave-3kk6 krb5kdc[30293](info): commencing operation
krb5kdc: starting...
W20260807 06:48:24.532294 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:24 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:24.779414 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:46041
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:46041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:25.132530 30307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:25.132853 30307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:25.133037 30307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:25.140413 30307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:25.140542 30307 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 06:48:25.140650 30307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:25.140770 30307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:25.140862 30307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:25.140980 30307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:25.151043 30307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:46041
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:25.152453 30307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:25.153875 30307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:25.166347 30313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:25.166424 30314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:25.166399 30316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:25.167570 30307 server_base.cc:1034] running on GCE node
I20260807 06:48:25.168406 30307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:25.169842 30307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:25.171090 30307 hybrid_clock.cc:648] HybridClock initialized: now 1786085305171012 us; error 56 us; skew 500 ppm
I20260807 06:48:25.171447 30307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:25 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:25.175042 30307 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:25.176524 30307 webserver.cc:460] Webserver started at http://127.27.54.126:39271/ using document root <none> and password file <none>
I20260807 06:48:25.177269 30307 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:25.177408 30307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:25.177744 30307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:25.180912 30307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "2e5ce310701542a6b9184d085fd7a91f"
format_stamp: "Formatted at 2026-08-07 06:48:25 on dist-test-slave-3kk6"
server_key: "01880902f90f7ace4b7325ddf26c7de5"
server_key_iv: "62a7d83d5f7941dd3d2a975b05bbd561"
server_key_version: "encryptionkey@0"
I20260807 06:48:25.181746 30307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "2e5ce310701542a6b9184d085fd7a91f"
format_stamp: "Formatted at 2026-08-07 06:48:25 on dist-test-slave-3kk6"
server_key: "01880902f90f7ace4b7325ddf26c7de5"
server_key_iv: "62a7d83d5f7941dd3d2a975b05bbd561"
server_key_version: "encryptionkey@0"
I20260807 06:48:25.187320 30307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:48:25.191045 30333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:25.192355 30307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:48:25.192567 30307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "2e5ce310701542a6b9184d085fd7a91f"
format_stamp: "Formatted at 2026-08-07 06:48:25 on dist-test-slave-3kk6"
server_key: "01880902f90f7ace4b7325ddf26c7de5"
server_key_iv: "62a7d83d5f7941dd3d2a975b05bbd561"
server_key_version: "encryptionkey@0"
I20260807 06:48:25.192831 30307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:25.212512 30307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:25.217412 30307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:25.217727 30307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:25.241201 30307 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:46041
I20260807 06:48:25.241223 30430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:46041 every 8 connection(s)
I20260807 06:48:25.242772 30307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:25.244073 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 30307
I20260807 06:48:25.244302 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:25.244972 27865 external_mini_cluster.cc:1428] Setting key 2ba22328d32550e461590ff7d84657cf
I20260807 06:48:25.247336 30432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:48:25 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:25.261253 30432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Bootstrap starting.
I20260807 06:48:25.264575 30432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:25.265669 30432 log.cc:824] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:25.268704 30432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: No bootstrap required, opened a new log
I20260807 06:48:25.268724 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:25.247095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.27.54.126:46041)
0807 06:48:25.247930 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:25.247943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:25.247992 (+ 49us) server_negotiation.cc:407] Connection header received
0807 06:48:25.249602 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:25.249623 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:25.250038 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:25.250378 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:25.252337 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.253745 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.255600 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.256259 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.256402 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:25.261335 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:25.261343 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:25.261357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:25.261426 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:25.264611 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:25.265354 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:25.265361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:25.265367 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:25.265549 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:25.265987 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:25.265991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:25.265993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:25.266510 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:25.266890 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:25.267317 (+ 427us) server_negotiation.cc:299] Negotiation successful
0807 06:48:25.267727 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":391,"threads_started":1}
I20260807 06:48:25.277182 30432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5ce310701542a6b9184d085fd7a91f" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46041 } }
I20260807 06:48:25.277602 30432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:25.277719 30432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5ce310701542a6b9184d085fd7a91f, State: Initialized, Role: FOLLOWER
I20260807 06:48:25.278285 30432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5ce310701542a6b9184d085fd7a91f" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46041 } }
I20260807 06:48:25.278537 30432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:25.278707 30432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:25.278905 30432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:25.281181 30432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5ce310701542a6b9184d085fd7a91f" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46041 } }
I20260807 06:48:25.281765 30432 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5ce310701542a6b9184d085fd7a91f; no voters:
I20260807 06:48:25.282572 30432 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:25.282797 30438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:25.283560 30438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [term 1 LEADER]: Becoming Leader. State: Replica: 2e5ce310701542a6b9184d085fd7a91f, State: Running, Role: LEADER
I20260807 06:48:25.284092 30438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5ce310701542a6b9184d085fd7a91f" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46041 } }
I20260807 06:48:25.284952 30432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:25.287194 30442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5ce310701542a6b9184d085fd7a91f. Latest consensus state: current_term: 1 leader_uuid: "2e5ce310701542a6b9184d085fd7a91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5ce310701542a6b9184d085fd7a91f" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46041 } } }
I20260807 06:48:25.287163 30440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e5ce310701542a6b9184d085fd7a91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5ce310701542a6b9184d085fd7a91f" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46041 } } }
I20260807 06:48:25.287637 30440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:25.287637 30442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:25.288564 30447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:25.293497 30447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:25.303197 30447 catalog_manager.cc:1347] Generated new cluster ID: 8d7b92d184b34b1c82dd2587dacb4e25
I20260807 06:48:25.303345 30447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:25.319475 30447 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:25.321422 30447 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:25.333421 30447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Generated new TSK 0
I20260807 06:48:25.334605 30447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:25.393811 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46041
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:25.808494 30471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:25.808851 30471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:25.809023 30471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:25.816522 30471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:25.816695 30471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:25.816843 30471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 06:48:25.816934 30471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:48:25.817070 30471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:25.827378 30471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:25.828897 30471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:25.830331 30471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:25.844975 30477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:25.845336 30476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:25.844967 30482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:25.846504 30471 server_base.cc:1034] running on GCE node
I20260807 06:48:25.847329 30471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:25.848547 30471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:25.849794 30471 hybrid_clock.cc:648] HybridClock initialized: now 1786085305849725 us; error 60 us; skew 500 ppm
I20260807 06:48:25.850171 30471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:25 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:25.854079 30471 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:25.856122 30471 webserver.cc:460] Webserver started at http://127.27.54.65:41651/ using document root <none> and password file <none>
I20260807 06:48:25.856932 30471 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:25.857106 30471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:25.857491 30471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:25.861094 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf"
format_stamp: "Formatted at 2026-08-07 06:48:25 on dist-test-slave-3kk6"
server_key: "7040922103758c518252e6cfeb03c019"
server_key_iv: "d0c2ad010257ca930a925d2a1a5c3668"
server_key_version: "encryptionkey@0"
I20260807 06:48:25.862022 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf"
format_stamp: "Formatted at 2026-08-07 06:48:25 on dist-test-slave-3kk6"
server_key: "7040922103758c518252e6cfeb03c019"
server_key_iv: "d0c2ad010257ca930a925d2a1a5c3668"
server_key_version: "encryptionkey@0"
I20260807 06:48:25.868340 30471 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 06:48:25.872702 30493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:25.874377 30471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:48:25.874603 30471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf"
format_stamp: "Formatted at 2026-08-07 06:48:25 on dist-test-slave-3kk6"
server_key: "7040922103758c518252e6cfeb03c019"
server_key_iv: "d0c2ad010257ca930a925d2a1a5c3668"
server_key_version: "encryptionkey@0"
I20260807 06:48:25.874907 30471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:25.894743 30471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:25.900085 30471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:25.900432 30471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:25.904053 30471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:25.904194 30471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:25.904349 30471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:25.904450 30471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:48:25 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:25.935501 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:25.906246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51571 (local address 127.27.54.126:46041)
0807 06:48:25.911130 (+ 4884us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:25.911136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:25.911159 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:48:25.911233 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:25.911241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:25.911365 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:25.914624 (+ 3259us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:25.914904 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.916304 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.919470 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.920193 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.920318 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:25.926105 (+ 5787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:25.926115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:25.926123 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:25.926180 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:25.930317 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:25.931709 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:25.931719 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:25.931725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:25.931909 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:25.932704 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:25.932722 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:25.932728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:25.933174 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:25.933369 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:25.934619 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0807 06:48:25.934972 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4708,"thread_start_us":4565,"threads_started":1}
I20260807 06:48:25.937295 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:25.905551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.27.54.65:51571)
0807 06:48:25.906580 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:25.906630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:25.909533 (+ 2903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:25.911988 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:25.912005 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:25.913113 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:25.914344 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:25.914387 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.916578 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.916591 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:25.919234 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:25.919257 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.920750 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.920755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:25.920954 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:25.921795 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:25.921852 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:25.925766 (+ 3914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:25.930518 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:25.930535 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:25.930565 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:25.931478 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:25.932099 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:25.932106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:25.932111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:25.932519 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:25.933362 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:25.933373 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:25.934002 (+ 629us) client_negotiation.cc:769] Sending connection context
0807 06:48:25.934473 (+ 471us) client_negotiation.cc:240] Negotiation successful
0807 06:48:25.935238 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":471,"threads_started":1}
I20260807 06:48:25.952785 30471 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:44731
I20260807 06:48:25.952800 30675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:44731 every 8 connection(s)
I20260807 06:48:25.954460 30471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:25.956848 30676 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46041
I20260807 06:48:25.957111 30676 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:25.957798 30676 heartbeater.cc:511] Master 127.27.54.126:46041 requested a full tablet report, sending...
I20260807 06:48:25.960151 30363 ts_manager.cc:194] Registered new tserver with Master: bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731)
I20260807 06:48:25.960662 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 30471
I20260807 06:48:25.960919 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:25.961771 27865 external_mini_cluster.cc:1428] Setting key 5a6ab80b295fa67ba878cce5c129ea33
I20260807 06:48:25.961872 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:25.945842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:35403 (local address 127.27.54.126:46041)
0807 06:48:25.946099 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:25.946109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:25.947251 (+ 1142us) server_negotiation.cc:407] Connection header received
0807 06:48:25.947517 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:25.947525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:25.947643 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:25.947785 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:25.948936 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.950272 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.951969 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.952738 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.952897 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:25.955133 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:25.955143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:25.955149 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:25.955191 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:25.958734 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:25.959432 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:25.959437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:25.959440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:25.959614 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:25.960022 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:25.960030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:25.960033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:25.960441 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:25.960628 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:25.961206 (+ 578us) server_negotiation.cc:299] Negotiation successful
0807 06:48:25.961481 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:48:25.961934 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:25.946322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.27.54.65:35403)
0807 06:48:25.947102 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:25.947132 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:25.947343 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:25.947838 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:25.947846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:25.948462 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:25.948774 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:25.948784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.950430 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.950434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:25.951799 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:25.951812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:25.952964 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:25.952973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:25.953169 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:25.953904 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:25.953923 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:25.954893 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:25.958859 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:25.958864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:25.958868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:25.959292 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:25.959745 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:25.959748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:25.959750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:25.959897 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:25.960635 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:25.960641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:25.960865 (+ 224us) client_negotiation.cc:769] Sending connection context
0807 06:48:25.961034 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 06:48:25.961414 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":470,"threads_started":1}
I20260807 06:48:25.962283 30363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:51571
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:26.018765 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46041
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:26.369645 30682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:26.369984 30682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:26.370173 30682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:26.377926 30682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:26.378068 30682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:26.378247 30682 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 06:48:26.378336 30682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:48:26.378468 30682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:26.389056 30682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:26.390511 30682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:26.391898 30682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:26.405682 30687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:26.405730 30689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:26.406190 30694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:26.407480 30682 server_base.cc:1034] running on GCE node
I20260807 06:48:26.408277 30682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:26.409466 30682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:26.410760 30682 hybrid_clock.cc:648] HybridClock initialized: now 1786085306410694 us; error 58 us; skew 500 ppm
I20260807 06:48:26.411227 30682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:26 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:26.415844 30682 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:26.417725 30682 webserver.cc:460] Webserver started at http://127.27.54.66:36305/ using document root <none> and password file <none>
I20260807 06:48:26.418694 30682 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:26.418845 30682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:26.419250 30682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:26.422511 30682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "395f258fde344497b06afaf8db3846ff"
format_stamp: "Formatted at 2026-08-07 06:48:26 on dist-test-slave-3kk6"
server_key: "a04f9c31ddbc47c6434670ba6e538f88"
server_key_iv: "345837a3988452220e8124213922b0c7"
server_key_version: "encryptionkey@0"
I20260807 06:48:26.423434 30682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "395f258fde344497b06afaf8db3846ff"
format_stamp: "Formatted at 2026-08-07 06:48:26 on dist-test-slave-3kk6"
server_key: "a04f9c31ddbc47c6434670ba6e538f88"
server_key_iv: "345837a3988452220e8124213922b0c7"
server_key_version: "encryptionkey@0"
I20260807 06:48:26.430145 30682 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 06:48:26.434839 30704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:26.437713 30682 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260807 06:48:26.437947 30682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "395f258fde344497b06afaf8db3846ff"
format_stamp: "Formatted at 2026-08-07 06:48:26 on dist-test-slave-3kk6"
server_key: "a04f9c31ddbc47c6434670ba6e538f88"
server_key_iv: "345837a3988452220e8124213922b0c7"
server_key_version: "encryptionkey@0"
I20260807 06:48:26.438234 30682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:26.459988 30682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:26.465061 30682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:26.465345 30682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:26.468485 30682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:26.468658 30682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:26.468793 30682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:26.468928 30682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:48:26 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:26.494390 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:26.468763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:60899 (local address 127.27.54.126:46041)
0807 06:48:26.471709 (+ 2946us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:26.471717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:26.471744 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:48:26.473153 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:26.473158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:26.473252 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:26.473387 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:26.475760 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.477143 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.480090 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.480732 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.480881 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:26.486567 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:26.486573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:26.486576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:26.486612 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:26.490288 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:26.491579 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:26.491583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:26.491585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:26.491750 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:26.492362 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:26.492365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:26.492367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:26.492638 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:26.492768 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:26.493918 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0807 06:48:26.494105 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2769,"thread_start_us":2662,"threads_started":1}
I20260807 06:48:26.496855 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:26.469582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.27.54.66:60899)
0807 06:48:26.470570 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:26.470623 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:26.472848 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:26.473655 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:26.473663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:26.474473 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:26.475477 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:26.475511 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.477351 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.477359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:26.479857 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:26.479879 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.480998 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.481006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:26.481335 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:26.482282 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:26.482331 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:26.486286 (+ 3955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:26.490494 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:26.490509 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:26.490535 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:26.491413 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:26.491876 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:26.491885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:26.491891 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:26.492232 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:26.492778 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:26.492790 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:26.493402 (+ 612us) client_negotiation.cc:769] Sending connection context
0807 06:48:26.493799 (+ 397us) client_negotiation.cc:240] Negotiation successful
0807 06:48:26.494578 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":388,"threads_started":1}
I20260807 06:48:26.521551 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:26.505498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:53293 (local address 127.27.54.126:46041)
0807 06:48:26.505819 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:26.505828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:26.507505 (+ 1677us) server_negotiation.cc:407] Connection header received
0807 06:48:26.507609 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:26.507619 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:26.507800 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:26.507944 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:26.508994 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.510316 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.512393 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.512841 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.512986 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:26.515097 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:26.515105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:26.515112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:26.515164 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:26.518720 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:26.519458 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:26.519463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:26.519467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:26.519600 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:26.519981 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:26.519986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:26.519990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:26.520328 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:26.520515 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:26.520947 (+ 432us) server_negotiation.cc:299] Negotiation successful
0807 06:48:26.521188 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260807 06:48:26.521660 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:26.505337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.27.54.66:53293)
0807 06:48:26.507240 (+ 1903us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:26.507265 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:26.507400 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:26.507997 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:26.508001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:26.508504 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:26.508823 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:26.508834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.510504 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.510513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:26.512218 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:26.512235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:26.513062 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:26.513069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:26.513230 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:26.513951 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:26.513969 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:26.514870 (+ 901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:26.518870 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:26.518878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:26.518886 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:26.519316 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:26.519714 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:26.519720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:26.519723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:26.519858 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:26.520456 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:26.520461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:26.520657 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 06:48:26.520819 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 06:48:26.521183 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1731,"thread_start_us":1563,"threads_started":1}
I20260807 06:48:26.529368 30682 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:38155
I20260807 06:48:26.529410 30896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:38155 every 8 connection(s)
I20260807 06:48:26.530740 30682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:26.533128 30899 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46041
I20260807 06:48:26.533334 30899 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:26.533998 30899 heartbeater.cc:511] Master 127.27.54.126:46041 requested a full tablet report, sending...
I20260807 06:48:26.535518 30362 ts_manager.cc:194] Registered new tserver with Master: 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155)
I20260807 06:48:26.536585 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 30682
I20260807 06:48:26.536821 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:26.536791 30362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:60899
I20260807 06:48:26.537382 27865 external_mini_cluster.cc:1428] Setting key 8a65b61bf7966dec696c5a904479a5a2
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:26.590960 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46041
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:26.952527 30904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:26.952821 30904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:26.952991 30904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:26.961012 30904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:26.961126 30904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:26.961302 30904 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 06:48:26.961386 30904 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:48:26.961517 30904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:26.966341 30676 heartbeater.cc:503] Master 127.27.54.126:46041 was elected leader, sending a full tablet report...
I20260807 06:48:26.972470 30904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:26.974046 30904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:26.975571 30904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:26.989897 30910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:26.989849 30913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:26.989938 30917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:26.990861 30904 server_base.cc:1034] running on GCE node
I20260807 06:48:26.991585 30904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:26.992591 30904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:26.993846 30904 hybrid_clock.cc:648] HybridClock initialized: now 1786085306993777 us; error 57 us; skew 500 ppm
I20260807 06:48:26.994196 30904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:26 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:26.997831 30904 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:26.999483 30904 webserver.cc:460] Webserver started at http://127.27.54.67:37785/ using document root <none> and password file <none>
I20260807 06:48:27.000212 30904 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:27.000402 30904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:27.000783 30904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:27.003930 30904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "3088a2d4967b44c7be0a90ce964ebd9c"
format_stamp: "Formatted at 2026-08-07 06:48:26 on dist-test-slave-3kk6"
server_key: "ec97b908b046b37389ce2f3d3b1c6083"
server_key_iv: "a59ca5145aebf64ca0ee17608be41cf4"
server_key_version: "encryptionkey@0"
I20260807 06:48:27.004743 30904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "3088a2d4967b44c7be0a90ce964ebd9c"
format_stamp: "Formatted at 2026-08-07 06:48:26 on dist-test-slave-3kk6"
server_key: "ec97b908b046b37389ce2f3d3b1c6083"
server_key_iv: "a59ca5145aebf64ca0ee17608be41cf4"
server_key_version: "encryptionkey@0"
I20260807 06:48:27.010078 30904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:48:27.013695 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:27.015142 30904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:27.015343 30904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "3088a2d4967b44c7be0a90ce964ebd9c"
format_stamp: "Formatted at 2026-08-07 06:48:26 on dist-test-slave-3kk6"
server_key: "ec97b908b046b37389ce2f3d3b1c6083"
server_key_iv: "a59ca5145aebf64ca0ee17608be41cf4"
server_key_version: "encryptionkey@0"
I20260807 06:48:27.015592 30904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:27.046193 30904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:27.051208 30904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:27.051555 30904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:27.054486 30904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:27.054639 30904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:27.054826 30904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:27.054934 30904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:48:27 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:27.082868 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.055448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:56427 (local address 127.27.54.126:46041)
0807 06:48:27.055983 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.055992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.057436 (+ 1444us) server_negotiation.cc:407] Connection header received
0807 06:48:27.060863 (+ 3427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.060870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.060984 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.061136 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:27.064241 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.065294 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.067999 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.068624 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.068776 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.075282 (+ 6506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:27.075290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:27.075295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:27.075354 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:27.079313 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.080182 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.080187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.080189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.080329 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.080839 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.080843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.080845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.081156 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:27.081324 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.082149 (+ 825us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.082409 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":235,"threads_started":1}
I20260807 06:48:27.084843 30945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.056270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.27.54.67:56427)
0807 06:48:27.057214 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.057273 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.060406 (+ 3133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.061496 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.061514 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.062688 (+ 1174us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:27.063956 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.063989 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.065563 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.065571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.067693 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.067744 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.068946 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.068957 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.069343 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.070312 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:27.070354 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:27.074879 (+ 4525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:27.079481 (+ 4602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:27.079490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:27.079510 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:27.080009 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:27.080455 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:27.080460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:27.080464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:27.080700 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:27.081312 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:27.081321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:27.081749 (+ 428us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.082035 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.082680 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":374,"threads_started":1}
I20260807 06:48:27.108372 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.091263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:35405 (local address 127.27.54.126:46041)
0807 06:48:27.091474 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.091482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.092316 (+ 834us) server_negotiation.cc:407] Connection header received
0807 06:48:27.092529 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.092533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.092601 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.092690 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:27.094298 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.095344 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.097611 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.098539 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.098791 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.101271 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:27.101276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:27.101279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:27.101314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:27.104694 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.105804 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.105817 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.105823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.106021 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.106590 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.106597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.106603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.107085 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:27.107264 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.107700 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.107964 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:48:27.108471 31046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.091358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.27.54.67:35405)
0807 06:48:27.092158 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.092187 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.092392 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.092770 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.092778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.093588 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:27.094105 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.094123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.095537 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.095541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.097431 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.097456 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.098814 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.098824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.099015 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.099775 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:27.099806 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:27.100991 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:27.104849 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:27.104858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:27.104863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:27.105560 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:27.106207 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:27.106218 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:27.106225 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:27.106434 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:27.107235 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:27.107252 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:27.107445 (+ 193us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.107600 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.107964 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":394,"threads_started":1}
I20260807 06:48:27.119495 30904 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:37011
I20260807 06:48:27.119517 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:37011 every 8 connection(s)
I20260807 06:48:27.121002 30904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:27.123322 31120 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46041
I20260807 06:48:27.123574 31120 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:27.124255 31120 heartbeater.cc:511] Master 127.27.54.126:46041 requested a full tablet report, sending...
I20260807 06:48:27.125725 30362 ts_manager.cc:194] Registered new tserver with Master: 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:27.126583 30362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:56427
I20260807 06:48:27.129361 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 30904
I20260807 06:48:27.129534 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:27.129948 27865 external_mini_cluster.cc:1428] Setting key c6bd93229a6c9959a3e4051711364aa9
I20260807 06:48:27.133042 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:48:27 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:48:27 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:27.191500 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.176434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.27.54.126:46041)
0807 06:48:27.176696 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.176704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.177070 (+ 366us) server_negotiation.cc:407] Connection header received
0807 06:48:27.177275 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.177279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.177364 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.177459 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:27.179086 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.179932 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.181140 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.181548 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.181652 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.186195 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:27.186200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:27.186203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:27.186238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:27.189215 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.189833 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.189837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.189839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.189964 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.190282 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.190285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.190287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.190570 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:27.190867 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.191069 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.191235 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 06:48:27.195789 30362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:27.203650 30362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:27.242959 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.220201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.27.54.67:37011)
0807 06:48:27.228671 (+ 8470us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.228684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.233438 (+ 4754us) server_negotiation.cc:407] Connection header received
0807 06:48:27.233682 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.233691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.233967 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.234190 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.235548 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.237332 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.239814 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.241428 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.241551 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.241775 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.242397 (+ 622us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.242581 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8203,"thread_start_us":7933,"threads_started":1}
I20260807 06:48:27.242947 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.216428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:38155 (local address 127.0.0.1:60016)
0807 06:48:27.232855 (+ 16427us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.232931 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.233682 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.234475 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.234484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.234511 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.235109 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.235123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.236552 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.236560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.238799 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.238815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.241713 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.241719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.242013 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.242075 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.242216 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.242458 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16058,"mutex_wait_us":1,"thread_start_us":15925,"threads_started":1}
I20260807 06:48:27.243636 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.216919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.27.54.65:44731)
0807 06:48:27.231727 (+ 14808us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.231750 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.233353 (+ 1603us) server_negotiation.cc:407] Connection header received
0807 06:48:27.233682 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.233691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.233967 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.234190 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.235673 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.237310 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.239824 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.241447 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.241594 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.241809 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.243094 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.243278 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14586,"thread_start_us":14253,"threads_started":1}
I20260807 06:48:27.243834 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.216632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.27.54.66:38155)
0807 06:48:27.232217 (+ 15585us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.232227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.233081 (+ 854us) server_negotiation.cc:407] Connection header received
0807 06:48:27.233880 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.233887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.234060 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.234675 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.235305 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.236406 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.238994 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.241161 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.241324 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.241564 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.243315 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.243535 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15428,"thread_start_us":15289,"threads_started":1}
I20260807 06:48:27.244105 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.219341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.0.0.1:47010)
0807 06:48:27.233306 (+ 13965us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.233322 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.233416 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.234223 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.234230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.234272 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.235313 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.235329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.237517 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.237522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.239565 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.239581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.241952 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.241960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.242136 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.242215 (+ 79us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.243598 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.243741 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13828,"mutex_wait_us":454,"thread_start_us":13783,"threads_started":1}
I20260807 06:48:27.244722 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.216568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44731 (local address 127.0.0.1:35974)
0807 06:48:27.233182 (+ 16614us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.233200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.233416 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.234582 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.234587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.234613 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.235490 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.235502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.237482 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.237488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.239590 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.239608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.241995 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.242001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.242628 (+ 627us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.242692 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.242792 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.242951 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16013,"mutex_wait_us":433,"thread_start_us":15854,"threads_started":1}
I20260807 06:48:27.248205 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d (DEFAULT_TABLE table=test-table [id=4f8b4975ca2b4f1a98a922d5449d4af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:27.248366 31009 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d (DEFAULT_TABLE table=test-table [id=4f8b4975ca2b4f1a98a922d5449d4af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:27.248374 30558 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d (DEFAULT_TABLE table=test-table [id=4f8b4975ca2b4f1a98a922d5449d4af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:27.249601 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.249756 31009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.249755 30558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.271499 31153 tablet_bootstrap.cc:492] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Bootstrap starting.
I20260807 06:48:27.274612 31154 tablet_bootstrap.cc:492] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Bootstrap starting.
I20260807 06:48:27.275019 31152 tablet_bootstrap.cc:492] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Bootstrap starting.
I20260807 06:48:27.277182 31153 tablet_bootstrap.cc:654] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.278281 31153 log.cc:824] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:27.278306 31154 tablet_bootstrap.cc:654] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.278751 31152 tablet_bootstrap.cc:654] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.279618 31154 log.cc:824] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:27.280086 31152 log.cc:824] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:27.281767 31153 tablet_bootstrap.cc:492] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: No bootstrap required, opened a new log
I20260807 06:48:27.282083 31153 ts_tablet_manager.cc:1397] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260807 06:48:27.282536 31154 tablet_bootstrap.cc:492] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: No bootstrap required, opened a new log
I20260807 06:48:27.282902 31154 ts_tablet_manager.cc:1397] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260807 06:48:27.283278 31152 tablet_bootstrap.cc:492] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: No bootstrap required, opened a new log
I20260807 06:48:27.283556 31152 ts_tablet_manager.cc:1397] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260807 06:48:27.291397 31154 raft_consensus.cc:348] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.291857 31154 raft_consensus.cc:374] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:27.291999 31154 raft_consensus.cc:729] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f258fde344497b06afaf8db3846ff, State: Initialized, Role: FOLLOWER
I20260807 06:48:27.292644 31154 consensus_queue.cc:260] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.293825 30899 heartbeater.cc:503] Master 127.27.54.126:46041 was elected leader, sending a full tablet report...
I20260807 06:48:27.293551 31153 raft_consensus.cc:348] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.294003 31153 raft_consensus.cc:374] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:27.294142 31153 raft_consensus.cc:729] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Initialized, Role: FOLLOWER
I20260807 06:48:27.294416 31152 raft_consensus.cc:348] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.294952 31152 raft_consensus.cc:374] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:27.294798 31153 consensus_queue.cc:260] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.295135 31152 raft_consensus.cc:729] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Initialized, Role: FOLLOWER
I20260807 06:48:27.295712 31154 ts_tablet_manager.cc:1428] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260807 06:48:27.296067 31152 consensus_queue.cc:260] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.299606 31153 ts_tablet_manager.cc:1428] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent starting tablet: real 0.017s user 0.014s sys 0.005s
I20260807 06:48:27.303520 31120 heartbeater.cc:503] Master 127.27.54.126:46041 was elected leader, sending a full tablet report...
I20260807 06:48:27.304567 31152 ts_tablet_manager.cc:1428] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent starting tablet: real 0.021s user 0.004s sys 0.016s
I20260807 06:48:27.315183 31161 raft_consensus.cc:482] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:27.315490 31161 raft_consensus.cc:504] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.317062 31161 leader_election.cc:288] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:27.320523 31162 raft_consensus.cc:482] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:27.320925 31162 raft_consensus.cc:504] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.323101 31162 leader_election.cc:288] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:27.326570 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.317667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:47545 (local address 127.27.54.65:44731)
0807 06:48:27.317901 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.317908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.318677 (+ 769us) server_negotiation.cc:407] Connection header received
0807 06:48:27.318761 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.318766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.318887 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.318985 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.319871 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.321046 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.323484 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.324937 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.325104 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.325216 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.326013 (+ 797us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.326210 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:48:27.327301 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.317527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:59801 (local address 127.27.54.67:37011)
0807 06:48:27.318048 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.318053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.318364 (+ 311us) server_negotiation.cc:407] Connection header received
0807 06:48:27.318726 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.318731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.318814 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.318922 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.319871 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.321017 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.323858 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.325079 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.325279 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.325388 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.326734 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.326932 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260807 06:48:27.327630 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.317529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44731 (local address 127.27.54.66:47545)
0807 06:48:27.318430 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.318446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.318558 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.318997 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.319001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.319017 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.319645 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.319663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.321243 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.321248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.323261 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.323282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.325496 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.325504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.325778 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.325826 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.326882 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.327091 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":162,"thread_start_us":544,"threads_started":1}
I20260807 06:48:27.328889 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.317376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.27.54.66:59801)
0807 06:48:27.318191 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.318216 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.318407 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.318962 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.318969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.318999 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.319721 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.319743 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.321173 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.321180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.323635 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.323653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.326178 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.326187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.326462 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.326515 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.328122 (+ 1607us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.328351 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":109,"thread_start_us":412,"threads_started":1}
I20260807 06:48:27.328786 30597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2d6a9c3b08458ca7dfd6f11e0fa06d" candidate_uuid: "395f258fde344497b06afaf8db3846ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" is_pre_election: true
I20260807 06:48:27.329381 30597 raft_consensus.cc:2454] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395f258fde344497b06afaf8db3846ff in term 0.
I20260807 06:48:27.330250 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2d6a9c3b08458ca7dfd6f11e0fa06d" candidate_uuid: "395f258fde344497b06afaf8db3846ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" is_pre_election: true
I20260807 06:48:27.330432 30706 leader_election.cc:302] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395f258fde344497b06afaf8db3846ff, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:27.330802 31051 raft_consensus.cc:2454] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395f258fde344497b06afaf8db3846ff in term 0.
I20260807 06:48:27.330998 31161 raft_consensus.cc:2781] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:27.331161 31161 raft_consensus.cc:482] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:27.331298 31161 raft_consensus.cc:3037] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:27.335505 31161 raft_consensus.cc:504] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.336994 30597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2d6a9c3b08458ca7dfd6f11e0fa06d" candidate_uuid: "395f258fde344497b06afaf8db3846ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf"
I20260807 06:48:27.337002 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2d6a9c3b08458ca7dfd6f11e0fa06d" candidate_uuid: "395f258fde344497b06afaf8db3846ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c"
I20260807 06:48:27.337230 30597 raft_consensus.cc:3037] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:27.337249 31051 raft_consensus.cc:3037] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:27.339051 31161 leader_election.cc:288] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [CANDIDATE]: Term 1 election: Requested vote from peers bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:27.340801 31051 raft_consensus.cc:2454] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395f258fde344497b06afaf8db3846ff in term 1.
I20260807 06:48:27.341147 30597 raft_consensus.cc:2454] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395f258fde344497b06afaf8db3846ff in term 1.
I20260807 06:48:27.341409 30713 leader_election.cc:302] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, 395f258fde344497b06afaf8db3846ff; no voters:
I20260807 06:48:27.341910 31161 raft_consensus.cc:2781] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:27.342476 31161 raft_consensus.cc:686] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 1 LEADER]: Becoming Leader. State: Replica: 395f258fde344497b06afaf8db3846ff, State: Running, Role: LEADER
I20260807 06:48:27.343219 31161 consensus_queue.cc:237] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.343930 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.325700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:39467 (local address 127.27.54.66:38155)
0807 06:48:27.325948 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.325956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.335757 (+ 9801us) server_negotiation.cc:407] Connection header received
0807 06:48:27.335947 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.335952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.336034 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.336125 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.337110 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.338184 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.341223 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.342622 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.342794 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.342863 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.343472 (+ 609us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.343601 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 06:48:27.343976 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.323589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:38155 (local address 127.27.54.65:39467)
0807 06:48:27.335624 (+ 12035us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.335646 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.335804 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.336294 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.336298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.336361 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.336883 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.336897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.338393 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.338398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.340997 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.341016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.343092 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.343100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.343286 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.343317 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.343430 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.343567 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11896,"mutex_wait_us":29,"thread_start_us":11644,"threads_started":1}
I20260807 06:48:27.344034 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.324250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:36781 (local address 127.27.54.67:37011)
0807 06:48:27.327590 (+ 3340us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.327598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.335530 (+ 7932us) server_negotiation.cc:407] Connection header received
0807 06:48:27.335881 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.335890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.335967 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.336068 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.336910 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.338143 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.341305 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.342613 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.342785 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.342859 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.343585 (+ 726us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.343716 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3207,"thread_start_us":5672,"threads_started":1}
I20260807 06:48:27.344758 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.323553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.27.54.65:36781)
0807 06:48:27.335402 (+ 11849us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.335427 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.335654 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.336111 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.336117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.336146 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.336699 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.336720 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.338339 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.338344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.341118 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.341137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.343149 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.343155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.343376 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.343411 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.344206 (+ 795us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.344349 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11578,"mutex_wait_us":259,"spinlock_wait_cycles":54144,"thread_start_us":11339,"threads_started":1}
I20260807 06:48:27.345403 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2d6a9c3b08458ca7dfd6f11e0fa06d" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff" is_pre_election: true
I20260807 06:48:27.346992 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2d6a9c3b08458ca7dfd6f11e0fa06d" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" is_pre_election: true
I20260807 06:48:27.347436 31051 raft_consensus.cc:2379] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in current term 1: Already voted for candidate 395f258fde344497b06afaf8db3846ff in this term.
I20260807 06:48:27.348228 30500 leader_election.cc:302] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters: 3088a2d4967b44c7be0a90ce964ebd9c, 395f258fde344497b06afaf8db3846ff
I20260807 06:48:27.349011 31162 raft_consensus.cc:2726] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 06:48:27.349671 30360 catalog_manager.cc:5495] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 395f258fde344497b06afaf8db3846ff (127.27.54.66). New cstate: current_term: 1 leader_uuid: "395f258fde344497b06afaf8db3846ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } health_report { overall_health: UNKNOWN } } }
W20260807 06:48:27.372885 31121 tablet.cc:2369] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 06:48:27 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085305, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:27.423096 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.409402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46041 (local address 127.0.0.1:59624)
0807 06:48:27.410098 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.410115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.410257 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.410566 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.410569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.411366 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:27.411648 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.411658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.412656 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.412660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.413621 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.413632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.414390 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.414395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.414512 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.415283 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:27.415302 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:27.417697 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:27.420717 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:27.420728 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:27.420734 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:27.421193 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:27.421510 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:27.421513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:27.421516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:27.421699 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:27.422178 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:27.422185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:27.422332 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.422473 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.422781 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":409,"threads_started":1}
I20260807 06:48:27.423198 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.409422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.27.54.126:46041)
0807 06:48:27.409665 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.409674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.410220 (+ 546us) server_negotiation.cc:407] Connection header received
0807 06:48:27.410368 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.410372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.410432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.410518 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:27.411783 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.412534 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.413745 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.414179 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.414273 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.417905 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:27.417910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:27.417913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:27.417947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:27.420575 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.421299 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.421303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.421305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.421416 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:27.421791 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:27.421795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:27.421797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:27.422060 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:27.422171 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.422557 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.422914 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":93}
I20260807 06:48:27.428344 30360 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:59624:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 06:48:27.429332 30360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:27.435992 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc80744176b43e29b3958f02d064d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b5feb61c86444f0b34495572c294566]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:48:27.435997 30558 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc80744176b43e29b3958f02d064d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b5feb61c86444f0b34495572c294566]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:48:27.436335 31009 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc80744176b43e29b3958f02d064d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b5feb61c86444f0b34495572c294566]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:48:27.437026 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc80744176b43e29b3958f02d064d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.437032 30558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc80744176b43e29b3958f02d064d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.437203 31009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc80744176b43e29b3958f02d064d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.443791 31152 tablet_bootstrap.cc:492] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: Bootstrap starting.
I20260807 06:48:27.444661 31153 tablet_bootstrap.cc:492] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Bootstrap starting.
I20260807 06:48:27.445178 31154 tablet_bootstrap.cc:492] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: Bootstrap starting.
I20260807 06:48:27.445673 31152 tablet_bootstrap.cc:654] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.446432 31153 tablet_bootstrap.cc:654] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.446820 31154 tablet_bootstrap.cc:654] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.448266 31152 tablet_bootstrap.cc:492] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: No bootstrap required, opened a new log
I20260807 06:48:27.448447 31152 ts_tablet_manager.cc:1397] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260807 06:48:27.449335 31153 tablet_bootstrap.cc:492] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: No bootstrap required, opened a new log
I20260807 06:48:27.449177 31154 tablet_bootstrap.cc:492] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: No bootstrap required, opened a new log
I20260807 06:48:27.449345 31152 raft_consensus.cc:348] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.449613 31152 raft_consensus.cc:374] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:27.449662 31153 ts_tablet_manager.cc:1397] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:48:27.449731 31152 raft_consensus.cc:729] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Initialized, Role: FOLLOWER
I20260807 06:48:27.449995 31152 consensus_queue.cc:260] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.449514 31154 ts_tablet_manager.cc:1397] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260807 06:48:27.450682 31152 ts_tablet_manager.cc:1428] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 06:48:27.450536 31153 raft_consensus.cc:348] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.450912 31154 raft_consensus.cc:348] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.451164 31154 raft_consensus.cc:374] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:27.451273 31154 raft_consensus.cc:729] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f258fde344497b06afaf8db3846ff, State: Initialized, Role: FOLLOWER
I20260807 06:48:27.450997 31163 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.451357 31153 raft_consensus.cc:374] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:27.451434 31163 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.451483 31153 raft_consensus.cc:729] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Initialized, Role: FOLLOWER
I20260807 06:48:27.451560 31154 consensus_queue.cc:260] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.451825 31153 consensus_queue.cc:260] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.452061 31161 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.452328 31161 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.452310 31162 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.452579 31162 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.452806 31154 ts_tablet_manager.cc:1428] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 06:48:27.453088 31153 ts_tablet_manager.cc:1428] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260807 06:48:27.457355 30678 tablet.cc:2369] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:48:27.533175 30900 tablet.cc:2369] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:27.699944 31163 raft_consensus.cc:482] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:27.700237 31163 raft_consensus.cc:504] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.701880 31163 leader_election.cc:288] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731), 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155)
I20260807 06:48:27.710726 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.702330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:44089 (local address 127.27.54.66:38155)
0807 06:48:27.702536 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.702543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.703218 (+ 675us) server_negotiation.cc:407] Connection header received
0807 06:48:27.703449 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.703455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.703539 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.703650 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.704359 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.705497 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.708421 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.709709 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.709891 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.709979 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.710290 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.710393 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:48:27.710863 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.702222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:38155 (local address 127.27.54.67:44089)
0807 06:48:27.703063 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.703086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.703298 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.703685 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.703690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.703738 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.704151 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.704165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.705645 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.705649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.708222 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.708241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.709890 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.709895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.710071 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.710113 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.710231 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.710397 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":273,"spinlock_wait_cycles":67840,"thread_start_us":483,"threads_started":1}
I20260807 06:48:27.711076 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.702402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:39591 (local address 127.27.54.65:44731)
0807 06:48:27.702578 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.702586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.703221 (+ 635us) server_negotiation.cc:407] Connection header received
0807 06:48:27.703449 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.703455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.703539 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.703650 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:27.704244 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.705491 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.708628 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.709973 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.710130 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.710183 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.710604 (+ 421us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.710794 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:48:27.711427 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.702222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44731 (local address 127.27.54.67:39591)
0807 06:48:27.703063 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.703086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.703260 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.703685 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.703690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.703733 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:27.704039 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.704059 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.705644 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.705649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.708222 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.708241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.710214 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.710222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.710426 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.710456 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.710558 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.710710 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":204,"spinlock_wait_cycles":1408,"thread_start_us":501,"threads_started":1}
I20260807 06:48:27.711678 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc80744176b43e29b3958f02d064d00" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff" is_pre_election: true
I20260807 06:48:27.712003 30812 raft_consensus.cc:2454] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 0.
I20260807 06:48:27.712306 30597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc80744176b43e29b3958f02d064d00" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" is_pre_election: true
I20260807 06:48:27.712599 30597 raft_consensus.cc:2454] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 0.
I20260807 06:48:27.712606 30934 leader_election.cc:302] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, 395f258fde344497b06afaf8db3846ff; no voters:
I20260807 06:48:27.713081 31163 raft_consensus.cc:2781] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:27.713215 31163 raft_consensus.cc:482] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:27.713328 31163 raft_consensus.cc:3037] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:27.715627 31163 raft_consensus.cc:504] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.716218 31163 leader_election.cc:288] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 election: Requested vote from peers bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731), 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155)
I20260807 06:48:27.716691 30597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc80744176b43e29b3958f02d064d00" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf"
I20260807 06:48:27.716794 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc80744176b43e29b3958f02d064d00" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff"
I20260807 06:48:27.716915 30597 raft_consensus.cc:3037] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:27.717095 30812 raft_consensus.cc:3037] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:27.719113 30597 raft_consensus.cc:2454] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 1.
I20260807 06:48:27.719610 30929 leader_election.cc:302] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:27.719719 30812 raft_consensus.cc:2454] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 1.
I20260807 06:48:27.720012 31163 raft_consensus.cc:2781] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:27.720499 31163 raft_consensus.cc:686] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 LEADER]: Becoming Leader. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Running, Role: LEADER
I20260807 06:48:27.721113 31163 consensus_queue.cc:237] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:27.724598 31192 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: TxnStatusTablet state changed. Reason: New leader 3088a2d4967b44c7be0a90ce964ebd9c. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.725075 31192 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:48:27.725853 31195 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 06:48:27.726155 31195 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 06:48:27.726109 30360 catalog_manager.cc:5495] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67). New cstate: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } health_report { overall_health: HEALTHY } } }
I20260807 06:48:27.746526 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.739926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.27.54.67:37011)
0807 06:48:27.740117 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.740123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.740143 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:48:27.740385 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.740391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.740497 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.740665 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:27.741135 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.741900 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.743942 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.744600 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.744717 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.745323 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.745897 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.746013 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.746192 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.746276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:48:27.746682 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.739799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.0.0.1:47026)
0807 06:48:27.740013 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.740038 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.740174 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.740713 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.740721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.740737 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:27.740965 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.740981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.742039 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.742045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.743786 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.743804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.744782 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.744789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.744998 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.745100 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.746027 (+ 927us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.746054 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.746152 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.746333 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 06:48:27.754554 30812 raft_consensus.cc:1261] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Refusing update from remote peer 3088a2d4967b44c7be0a90ce964ebd9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:27.754626 30597 raft_consensus.cc:1261] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Refusing update from remote peer 3088a2d4967b44c7be0a90ce964ebd9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:27.755571 31192 consensus_queue.cc:1035] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:27.755946 31163 consensus_queue.cc:1035] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:27.766296 31161 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: TxnStatusTablet state changed. Reason: New leader 3088a2d4967b44c7be0a90ce964ebd9c. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.766394 31162 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: TxnStatusTablet state changed. Reason: New leader 3088a2d4967b44c7be0a90ce964ebd9c. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.766718 31161 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.766738 31162 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.772323 31200 mvcc.cc:205] Tried to move back new op lower bound from 7315805420552773632 to 7315805420429950976. Current Snapshot: MvccSnapshot[applied={T|T < 7315805420552773632}]
I20260807 06:48:27.780782 31162 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.781055 31162 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P bffbb70cb1cd4b749ca5bbf43a7dffaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.785614 31205 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.786053 31205 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:48:27.786691 31163 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.786969 31163 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 3088a2d4967b44c7be0a90ce964ebd9c: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:48:27.787266 31201 mvcc.cc:205] Tried to move back new op lower bound from 7315805420552773632 to 7315805420429950976. Current Snapshot: MvccSnapshot[applied={T|T < 7315805420552773632}]
I20260807 06:48:27.788683 31161 tablet_replica.cc:439] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } } }
I20260807 06:48:27.789057 31161 tablet_replica.cc:442] T 6bc80744176b43e29b3958f02d064d00 P 395f258fde344497b06afaf8db3846ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:48:27.804260 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.797019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.27.54.66:38155)
0807 06:48:27.797285 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.797294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.797839 (+ 545us) server_negotiation.cc:407] Connection header received
0807 06:48:27.798111 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.798115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.798196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.798342 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:27.798908 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.799835 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.801944 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.802358 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.802525 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.803051 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.803598 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.803720 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.803870 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.803941 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 06:48:27.822156 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.815800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:46807 (local address 127.27.54.67:37011)
0807 06:48:27.816015 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.816021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.816424 (+ 403us) server_negotiation.cc:407] Connection header received
0807 06:48:27.816619 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.816623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.816690 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.816777 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:27.817335 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.818120 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.820261 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.820811 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.820914 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.821499 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.821604 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.821674 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.821838 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.821909 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 06:48:27.822633 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.815598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.27.54.66:46807)
0807 06:48:27.816289 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.816331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.816494 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.816817 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.816822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.816835 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:27.817181 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.817192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.818264 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.818270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.820109 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.820128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.820998 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.821005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.821197 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.821273 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.821709 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.821721 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.821811 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.821967 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":323,"threads_started":1}
I20260807 06:48:27.833380 30597 raft_consensus.cc:1261] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Refusing update from remote peer 395f258fde344497b06afaf8db3846ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:27.833456 31051 raft_consensus.cc:1261] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Refusing update from remote peer 395f258fde344497b06afaf8db3846ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:27.834167 31174 consensus_queue.cc:1035] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:27.834508 31161 consensus_queue.cc:1035] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:27.872836 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.865198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:57915 (local address 127.27.54.66:38155)
0807 06:48:27.865441 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:27.865451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:27.866087 (+ 636us) server_negotiation.cc:407] Connection header received
0807 06:48:27.866312 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:27.866317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:27.866406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:27.866512 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:27.867307 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.868300 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.870489 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.871144 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.871284 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.872020 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.872143 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.872218 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:27.872513 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 06:48:27.872597 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 06:48:27.873219 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:27.865173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:38155 (local address 127.27.54.67:57915)
0807 06:48:27.865984 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:27.866006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:27.866163 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:27.866549 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:27.866554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:27.866567 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:27.867110 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.867123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.868443 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.868450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.870326 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:27.870345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:27.871362 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:27.871370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:27.871563 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:27.871660 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:27.872252 (+ 592us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:27.872264 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:48:27.872362 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:48:27.872522 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":429,"threads_started":1}
I20260807 06:48:27.920578 30360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:27.920895 30360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:27.924571 30360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Sending DeleteTablet for 3 replicas of tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d
I20260807 06:48:27.925753 30780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:27 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60016
I20260807 06:48:27.926328 30558 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:27 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:35974
I20260807 06:48:27.926851 31227 tablet_replica.cc:330] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: stopping tablet replica
I20260807 06:48:27.927127 31009 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:27 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47010
I20260807 06:48:27.927503 31227 raft_consensus.cc:2229] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:27.928380 31227 raft_consensus.cc:2258] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:27.930814 31227 ts_tablet_manager.cc:1905] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:27.933514 30360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:27.934306 30360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:27.941406 31227 ts_tablet_manager.cc:1918] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:27.941784 31227 log.cc:1196] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/2c2d6a9c3b08458ca7dfd6f11e0fa06d
I20260807 06:48:27.942587 31227 ts_tablet_manager.cc:1939] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 395f258fde344497b06afaf8db3846ff: Deleting consensus metadata
I20260807 06:48:27.944819 30337 catalog_manager.cc:4841] TS 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155): tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d (table test-table [id=4f8b4975ca2b4f1a98a922d5449d4af9]) successfully deleted
I20260807 06:48:27.949397 31006 tablet_service.cc:1467] Processing CreateTablet for tablet e3b95e7862b94cbbb61ff472acb44b53 (DEFAULT_TABLE table=test-table [id=8c233f1076a54629a61c47dd0394ae1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:27.949538 30568 tablet_service.cc:1467] Processing CreateTablet for tablet e3b95e7862b94cbbb61ff472acb44b53 (DEFAULT_TABLE table=test-table [id=8c233f1076a54629a61c47dd0394ae1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:27.950018 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b95e7862b94cbbb61ff472acb44b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.950037 30568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b95e7862b94cbbb61ff472acb44b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.960512 30780 tablet_service.cc:1467] Processing CreateTablet for tablet e3b95e7862b94cbbb61ff472acb44b53 (DEFAULT_TABLE table=test-table [id=8c233f1076a54629a61c47dd0394ae1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:27.961154 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b95e7862b94cbbb61ff472acb44b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:27.983180 31232 tablet_bootstrap.cc:492] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Bootstrap starting.
I20260807 06:48:27.991410 31233 tablet_bootstrap.cc:492] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Bootstrap starting.
I20260807 06:48:27.993713 31233 tablet_bootstrap.cc:654] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.994427 31229 tablet_replica.cc:330] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: stopping tablet replica
I20260807 06:48:27.998049 31232 tablet_bootstrap.cc:654] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:27.998740 31229 raft_consensus.cc:2229] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:27.999698 31229 raft_consensus.cc:2258] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:27.999363 31228 tablet_replica.cc:330] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: stopping tablet replica
I20260807 06:48:28.001459 31228 raft_consensus.cc:2229] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:28.001865 31228 raft_consensus.cc:2258] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:28.002298 31229 ts_tablet_manager.cc:1905] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:28.003736 31228 ts_tablet_manager.cc:1905] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:28.011247 31230 tablet_bootstrap.cc:492] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Bootstrap starting.
I20260807 06:48:28.012329 31232 tablet_bootstrap.cc:492] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: No bootstrap required, opened a new log
I20260807 06:48:28.012588 31232 ts_tablet_manager.cc:1397] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.009s
I20260807 06:48:28.013283 31230 tablet_bootstrap.cc:654] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:28.013643 31232 raft_consensus.cc:348] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.014024 31232 raft_consensus.cc:374] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:28.014284 31232 raft_consensus.cc:729] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f258fde344497b06afaf8db3846ff, State: Initialized, Role: FOLLOWER
I20260807 06:48:28.014698 31232 consensus_queue.cc:260] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.015750 31233 tablet_bootstrap.cc:492] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: No bootstrap required, opened a new log
I20260807 06:48:28.015969 31233 ts_tablet_manager.cc:1397] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20260807 06:48:28.016824 31233 raft_consensus.cc:348] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.017149 31233 raft_consensus.cc:374] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:28.017292 31233 raft_consensus.cc:729] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Initialized, Role: FOLLOWER
I20260807 06:48:28.017627 31233 consensus_queue.cc:260] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.018033 31232 ts_tablet_manager.cc:1428] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260807 06:48:28.020815 31233 ts_tablet_manager.cc:1428] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 06:48:28.024652 31229 ts_tablet_manager.cc:1918] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:28.026342 31228 ts_tablet_manager.cc:1918] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:28.026615 31228 log.cc:1196] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/2c2d6a9c3b08458ca7dfd6f11e0fa06d
I20260807 06:48:28.027331 31228 ts_tablet_manager.cc:1939] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting consensus metadata
I20260807 06:48:28.029529 30336 catalog_manager.cc:4841] TS bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731): tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d (table test-table [id=4f8b4975ca2b4f1a98a922d5449d4af9]) successfully deleted
I20260807 06:48:28.030086 31229 log.cc:1196] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/2c2d6a9c3b08458ca7dfd6f11e0fa06d
I20260807 06:48:28.028648 31230 tablet_bootstrap.cc:492] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: No bootstrap required, opened a new log
I20260807 06:48:28.030875 31229 ts_tablet_manager.cc:1939] T 2c2d6a9c3b08458ca7dfd6f11e0fa06d P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting consensus metadata
I20260807 06:48:28.033018 30338 catalog_manager.cc:4841] TS 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011): tablet 2c2d6a9c3b08458ca7dfd6f11e0fa06d (table test-table [id=4f8b4975ca2b4f1a98a922d5449d4af9]) successfully deleted
I20260807 06:48:28.030900 31230 ts_tablet_manager.cc:1397] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260807 06:48:28.034250 31230 raft_consensus.cc:348] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.035207 31230 raft_consensus.cc:374] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:28.035354 31230 raft_consensus.cc:729] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Initialized, Role: FOLLOWER
I20260807 06:48:28.035707 31230 consensus_queue.cc:260] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.039009 31230 ts_tablet_manager.cc:1428] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260807 06:48:28.064294 31162 raft_consensus.cc:482] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:28.064528 31162 raft_consensus.cc:504] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.065271 31162 leader_election.cc:288] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:28.065850 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b95e7862b94cbbb61ff472acb44b53" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff" is_pre_election: true
I20260807 06:48:28.066171 30812 raft_consensus.cc:2454] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 0.
I20260807 06:48:28.066196 31049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b95e7862b94cbbb61ff472acb44b53" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" is_pre_election: true
I20260807 06:48:28.066576 31049 raft_consensus.cc:2454] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 0.
I20260807 06:48:28.066828 30499 leader_election.cc:302] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395f258fde344497b06afaf8db3846ff, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:28.067173 31162 raft_consensus.cc:2781] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:28.067309 31162 raft_consensus.cc:482] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:28.067428 31162 raft_consensus.cc:3037] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:28.069727 31162 raft_consensus.cc:504] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.070483 31162 leader_election.cc:288] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 election: Requested vote from peers 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:28.071061 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b95e7862b94cbbb61ff472acb44b53" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff"
I20260807 06:48:28.071350 30812 raft_consensus.cc:3037] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:28.071362 31049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b95e7862b94cbbb61ff472acb44b53" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c"
I20260807 06:48:28.071590 31049 raft_consensus.cc:3037] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:28.074944 30812 raft_consensus.cc:2454] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 1.
I20260807 06:48:28.075224 31049 raft_consensus.cc:2454] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 1.
I20260807 06:48:28.075700 30499 leader_election.cc:302] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, 395f258fde344497b06afaf8db3846ff, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:28.076172 31162 raft_consensus.cc:2781] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:28.076735 31162 raft_consensus.cc:686] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 LEADER]: Becoming Leader. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Running, Role: LEADER
I20260807 06:48:28.077296 31162 consensus_queue.cc:237] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:28.081198 30362 catalog_manager.cc:5495] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf reported cstate change: term changed from 0 to 1, leader changed from <none> to bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65). New cstate: current_term: 1 leader_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } health_report { overall_health: UNKNOWN } } }
I20260807 06:48:28.135610 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:28.127791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.27.54.65:44731)
0807 06:48:28.127986 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:28.127992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:28.128008 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:48:28.128120 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:28.128126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:28.128225 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:28.128371 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:28.128950 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.130089 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.132150 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.132847 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.132966 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:28.133586 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:28.134401 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:28.134541 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:28.135197 (+ 656us) server_negotiation.cc:299] Negotiation successful
0807 06:48:28.135288 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 06:48:28.184688 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:28.175086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:49563 (local address 127.27.54.67:37011)
0807 06:48:28.175312 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:28.175318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:28.179144 (+ 3826us) server_negotiation.cc:407] Connection header received
0807 06:48:28.179504 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:28.179515 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:28.179601 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:28.179675 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:28.180425 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.181192 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.182600 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.183104 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.183268 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:28.183773 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:28.183888 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:28.183997 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:28.184147 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 06:48:28.184239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:48:28.184777 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:28.175012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.27.54.65:49563)
0807 06:48:28.179003 (+ 3991us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:28.179023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:28.179277 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:28.179908 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:28.179911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:28.179921 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:28.180256 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:28.180268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.181343 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.181350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:28.182439 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:28.182452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.183317 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.183325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:28.183507 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:28.183571 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:28.184004 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:28.184018 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 06:48:28.184121 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 06:48:28.184334 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3817,"thread_start_us":3613,"threads_started":1}
I20260807 06:48:28.197163 30812 raft_consensus.cc:1261] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Refusing update from remote peer bffbb70cb1cd4b749ca5bbf43a7dffaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:28.197839 31049 raft_consensus.cc:1261] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Refusing update from remote peer bffbb70cb1cd4b749ca5bbf43a7dffaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:28.198987 31238 consensus_queue.cc:1035] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:28.199889 31238 consensus_queue.cc:1035] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:28.218097 31198 mvcc.cc:205] Tried to move back new op lower bound from 7315805422368215040 to 7315805421889048576. Current Snapshot: MvccSnapshot[applied={T|T < 7315805422368215040}]
I20260807 06:48:28.252419 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:28.244706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44731 (local address 127.27.54.67:43485)
0807 06:48:28.245114 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:28.245152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:28.245300 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:28.246580 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:28.246583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:28.246594 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:28.246837 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:28.246848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.248556 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.248561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:28.250093 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:28.250108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.251114 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.251119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:28.251303 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:28.251362 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:28.251906 (+ 544us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:28.251918 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:48:28.252001 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 06:48:28.252163 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260807 06:48:28.253285 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:28.245935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:43485 (local address 127.27.54.65:44731)
0807 06:48:28.246125 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:28.246131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:28.246150 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:48:28.246219 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:28.246224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:28.246313 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:28.246418 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:28.247327 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.248377 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.250256 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:28.250802 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:28.250976 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:28.251565 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:28.251741 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:28.251848 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:28.252940 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0807 06:48:28.253031 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 06:48:28.254601 31200 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 06:48:29.880411 30362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:29.880667 30362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:29.885641 30362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Sending DeleteTablet for 3 replicas of tablet e3b95e7862b94cbbb61ff472acb44b53
I20260807 06:48:29.886874 30558 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b95e7862b94cbbb61ff472acb44b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:29 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:35974
I20260807 06:48:29.886863 30780 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b95e7862b94cbbb61ff472acb44b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:29 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60016
I20260807 06:48:29.887629 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b95e7862b94cbbb61ff472acb44b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:29 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47010
I20260807 06:48:29.887791 31277 tablet_replica.cc:330] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: stopping tablet replica
I20260807 06:48:29.888111 31277 raft_consensus.cc:2229] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:29.888860 31277 raft_consensus.cc:2258] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:29.890210 31277 ts_tablet_manager.cc:1905] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:29.897045 31279 tablet_replica.cc:330] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: stopping tablet replica
I20260807 06:48:29.897758 31279 raft_consensus.cc:2229] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:29.898205 31279 raft_consensus.cc:2258] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:29.899711 31279 ts_tablet_manager.cc:1905] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:29.899827 31277 ts_tablet_manager.cc:1918] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:48:29.900098 31277 log.cc:1196] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/e3b95e7862b94cbbb61ff472acb44b53
I20260807 06:48:29.900779 31277 ts_tablet_manager.cc:1939] T e3b95e7862b94cbbb61ff472acb44b53 P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting consensus metadata
I20260807 06:48:29.902292 30336 catalog_manager.cc:4841] TS bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731): tablet e3b95e7862b94cbbb61ff472acb44b53 (table test-table [id=8c233f1076a54629a61c47dd0394ae1c]) successfully deleted
I20260807 06:48:29.907169 31278 tablet_replica.cc:330] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: stopping tablet replica
I20260807 06:48:29.907434 31278 raft_consensus.cc:2229] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:29.907749 31278 raft_consensus.cc:2258] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:29.908815 31278 ts_tablet_manager.cc:1905] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:29.910595 31279 ts_tablet_manager.cc:1918] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:48:29.910808 31279 log.cc:1196] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/e3b95e7862b94cbbb61ff472acb44b53
I20260807 06:48:29.911324 31279 ts_tablet_manager.cc:1939] T e3b95e7862b94cbbb61ff472acb44b53 P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting consensus metadata
I20260807 06:48:29.912734 30338 catalog_manager.cc:4841] TS 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011): tablet e3b95e7862b94cbbb61ff472acb44b53 (table test-table [id=8c233f1076a54629a61c47dd0394ae1c]) successfully deleted
I20260807 06:48:29.914296 31278 ts_tablet_manager.cc:1918] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:48:29.914477 31278 log.cc:1196] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/e3b95e7862b94cbbb61ff472acb44b53
I20260807 06:48:29.915017 31278 ts_tablet_manager.cc:1939] T e3b95e7862b94cbbb61ff472acb44b53 P 395f258fde344497b06afaf8db3846ff: Deleting consensus metadata
I20260807 06:48:29.916117 30337 catalog_manager.cc:4841] TS 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155): tablet e3b95e7862b94cbbb61ff472acb44b53 (table test-table [id=8c233f1076a54629a61c47dd0394ae1c]) successfully deleted
Aug 07 06:48:29 dist-test-slave-3kk6 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:48:29 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:29.968395 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:29.945477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.27.54.126:46041)
0807 06:48:29.946239 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:29.946243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:29.946269 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:48:29.946406 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:29.946411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:29.946485 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:29.946590 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:29.948257 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:29.949203 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:29.950628 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:29.951235 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:29.951364 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:29.957271 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:29.957281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:29.957287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:29.957347 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:29.962579 (+ 5232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:29.964141 (+ 1562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:29.964151 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:29.964157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:29.964431 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:29.965148 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:29.965156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:29.965162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:29.965716 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:29.966089 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:29.967542 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0807 06:48:29.967871 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":450,"threads_started":1}
I20260807 06:48:29.973155 30362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:29.973789 30362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:29.982867 31006 tablet_service.cc:1467] Processing CreateTablet for tablet 5d32bf00935f47df80ad0f7984ac69af (DEFAULT_TABLE table=test-table [id=63ff63feda634f2cbbac4923f6c4caed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:29.983340 30558 tablet_service.cc:1467] Processing CreateTablet for tablet 5d32bf00935f47df80ad0f7984ac69af (DEFAULT_TABLE table=test-table [id=63ff63feda634f2cbbac4923f6c4caed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:29.983489 31006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d32bf00935f47df80ad0f7984ac69af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:29.983832 30558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d32bf00935f47df80ad0f7984ac69af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:29.990698 31291 tablet_bootstrap.cc:492] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Bootstrap starting.
I20260807 06:48:29.990582 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 5d32bf00935f47df80ad0f7984ac69af (DEFAULT_TABLE table=test-table [id=63ff63feda634f2cbbac4923f6c4caed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:29.991190 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d32bf00935f47df80ad0f7984ac69af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:29.993316 31291 tablet_bootstrap.cc:654] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:29.997960 31291 tablet_bootstrap.cc:492] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: No bootstrap required, opened a new log
I20260807 06:48:29.998165 31291 ts_tablet_manager.cc:1397] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 06:48:29.998256 31294 tablet_bootstrap.cc:492] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Bootstrap starting.
I20260807 06:48:29.999166 31291 raft_consensus.cc:348] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:29.999545 31291 raft_consensus.cc:374] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:29.999692 31291 raft_consensus.cc:729] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Initialized, Role: FOLLOWER
I20260807 06:48:30.000010 31291 consensus_queue.cc:260] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.000771 31291 ts_tablet_manager.cc:1428] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 06:48:30.001581 31297 tablet_bootstrap.cc:492] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Bootstrap starting.
I20260807 06:48:30.002102 31294 tablet_bootstrap.cc:654] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:30.003479 31297 tablet_bootstrap.cc:654] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:30.005342 31294 tablet_bootstrap.cc:492] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: No bootstrap required, opened a new log
I20260807 06:48:30.005553 31294 ts_tablet_manager.cc:1397] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260807 06:48:30.006572 31294 raft_consensus.cc:348] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.006910 31294 raft_consensus.cc:374] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:30.007032 31294 raft_consensus.cc:729] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Initialized, Role: FOLLOWER
I20260807 06:48:30.007354 31297 tablet_bootstrap.cc:492] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: No bootstrap required, opened a new log
I20260807 06:48:30.007526 31297 ts_tablet_manager.cc:1397] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260807 06:48:30.007345 31294 consensus_queue.cc:260] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.008424 31294 ts_tablet_manager.cc:1428] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:30.008397 31297 raft_consensus.cc:348] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.008749 31297 raft_consensus.cc:374] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:30.008869 31297 raft_consensus.cc:729] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f258fde344497b06afaf8db3846ff, State: Initialized, Role: FOLLOWER
I20260807 06:48:30.009166 31297 consensus_queue.cc:260] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.010793 31297 ts_tablet_manager.cc:1428] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 06:48:30.107854 31262 raft_consensus.cc:482] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:30.108081 31262 raft_consensus.cc:504] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.108803 31262 leader_election.cc:288] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731), 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155)
I20260807 06:48:30.109462 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d32bf00935f47df80ad0f7984ac69af" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" is_pre_election: true
I20260807 06:48:30.109503 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d32bf00935f47df80ad0f7984ac69af" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff" is_pre_election: true
I20260807 06:48:30.109752 30596 raft_consensus.cc:2454] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 0.
I20260807 06:48:30.109833 30812 raft_consensus.cc:2454] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 0.
I20260807 06:48:30.110319 30929 leader_election.cc:302] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:30.110631 31262 raft_consensus.cc:2781] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:30.110868 31262 raft_consensus.cc:482] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:30.110989 31262 raft_consensus.cc:3037] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:30.113366 31262 raft_consensus.cc:504] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.113976 31262 leader_election.cc:288] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 election: Requested vote from peers bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731), 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155)
I20260807 06:48:30.114672 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d32bf00935f47df80ad0f7984ac69af" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff"
I20260807 06:48:30.114704 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d32bf00935f47df80ad0f7984ac69af" candidate_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf"
I20260807 06:48:30.114930 30596 raft_consensus.cc:3037] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:30.114964 30812 raft_consensus.cc:3037] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:30.118240 30812 raft_consensus.cc:2454] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 1.
I20260807 06:48:30.118253 30596 raft_consensus.cc:2454] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3088a2d4967b44c7be0a90ce964ebd9c in term 1.
I20260807 06:48:30.118881 30929 leader_election.cc:302] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:30.119290 31262 raft_consensus.cc:2781] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:30.119441 31262 raft_consensus.cc:686] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 LEADER]: Becoming Leader. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Running, Role: LEADER
I20260807 06:48:30.119769 31262 consensus_queue.cc:237] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.123080 30362 catalog_manager.cc:5495] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67). New cstate: current_term: 1 leader_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } health_report { overall_health: HEALTHY } } }
I20260807 06:48:30.171201 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:30.165637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.27.54.67:37011)
0807 06:48:30.166341 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:30.166345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:30.166372 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:48:30.166433 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:30.166436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:30.166505 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:30.166596 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:30.167075 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.167900 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.169264 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.169758 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.169858 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:30.170352 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:30.170484 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:30.170556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:30.170772 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 06:48:30.170870 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":448,"threads_started":1}
I20260807 06:48:30.186951 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:30.181078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:60061 (local address 127.27.54.67:37011)
0807 06:48:30.181307 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:30.181312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:30.181568 (+ 256us) server_negotiation.cc:407] Connection header received
0807 06:48:30.181773 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:30.181777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:30.181844 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:30.181928 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:30.182454 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.183267 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.184990 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.185490 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.185585 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:30.186080 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:30.186186 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:30.186259 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:30.186441 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 06:48:30.186545 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 06:48:30.187117 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:30.180741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:37011 (local address 127.27.54.67:60061)
0807 06:48:30.181473 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:30.181490 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:30.181614 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:30.181964 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:30.181967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:30.181976 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:48:30.182280 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:30.182288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.183421 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.183425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:30.184836 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:30.184849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.185665 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.185669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:30.185879 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:30.185919 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:48:30.186306 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:48:30.186318 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:48:30.186410 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 06:48:30.186550 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":409,"threads_started":1}
I20260807 06:48:30.196365 30596 raft_consensus.cc:1261] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Refusing update from remote peer 3088a2d4967b44c7be0a90ce964ebd9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:30.196373 30812 raft_consensus.cc:1261] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Refusing update from remote peer 3088a2d4967b44c7be0a90ce964ebd9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:30.197082 31253 consensus_queue.cc:1035] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:30.197448 31262 consensus_queue.cc:1035] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:30.250856 30363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:30.251138 30363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:30.254505 30363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Sending DeleteTablet for 3 replicas of tablet 5d32bf00935f47df80ad0f7984ac69af
I20260807 06:48:30.255556 30558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d32bf00935f47df80ad0f7984ac69af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:30 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:35974
I20260807 06:48:30.255607 30780 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d32bf00935f47df80ad0f7984ac69af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:30 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60016
I20260807 06:48:30.255918 31277 tablet_replica.cc:330] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: stopping tablet replica
I20260807 06:48:30.256161 31277 raft_consensus.cc:2229] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:30.256156 31009 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d32bf00935f47df80ad0f7984ac69af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:30 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47010
I20260807 06:48:30.256414 31277 raft_consensus.cc:2258] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:30.256454 31279 tablet_replica.cc:330] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: stopping tablet replica
I20260807 06:48:30.256736 31278 tablet_replica.cc:330] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: stopping tablet replica
I20260807 06:48:30.256732 31279 raft_consensus.cc:2229] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:30.256973 31278 raft_consensus.cc:2229] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:30.257201 31278 raft_consensus.cc:2258] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:30.257341 31279 raft_consensus.cc:2258] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:30.258412 31277 ts_tablet_manager.cc:1905] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:30.259399 31279 ts_tablet_manager.cc:1905] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:30.263083 31278 ts_tablet_manager.cc:1905] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:30.265918 31277 ts_tablet_manager.cc:1918] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:30.266101 31277 log.cc:1196] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/5d32bf00935f47df80ad0f7984ac69af
I20260807 06:48:30.266575 31277 ts_tablet_manager.cc:1939] T 5d32bf00935f47df80ad0f7984ac69af P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting consensus metadata
I20260807 06:48:30.266860 31279 ts_tablet_manager.cc:1918] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:48:30.267035 31279 log.cc:1196] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/5d32bf00935f47df80ad0f7984ac69af
I20260807 06:48:30.267530 31279 ts_tablet_manager.cc:1939] T 5d32bf00935f47df80ad0f7984ac69af P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting consensus metadata
I20260807 06:48:30.268478 30336 catalog_manager.cc:4841] TS bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731): tablet 5d32bf00935f47df80ad0f7984ac69af (table test-table [id=63ff63feda634f2cbbac4923f6c4caed]) successfully deleted
I20260807 06:48:30.268815 30338 catalog_manager.cc:4841] TS 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011): tablet 5d32bf00935f47df80ad0f7984ac69af (table test-table [id=63ff63feda634f2cbbac4923f6c4caed]) successfully deleted
I20260807 06:48:30.269856 30363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:48:30.270596 31278 ts_tablet_manager.cc:1918] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260807 06:48:30.270704 30363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:30.270850 31278 log.cc:1196] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/5d32bf00935f47df80ad0f7984ac69af
I20260807 06:48:30.271330 31278 ts_tablet_manager.cc:1939] T 5d32bf00935f47df80ad0f7984ac69af P 395f258fde344497b06afaf8db3846ff: Deleting consensus metadata
I20260807 06:48:30.272765 30337 catalog_manager.cc:4841] TS 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155): tablet 5d32bf00935f47df80ad0f7984ac69af (table test-table [id=63ff63feda634f2cbbac4923f6c4caed]) successfully deleted
I20260807 06:48:30.279536 30558 tablet_service.cc:1467] Processing CreateTablet for tablet 9a808961b4f44115b8f5ff5ae3295cbf (DEFAULT_TABLE table=test-table [id=dc432ac6a2ca46d1bfb5a7bc9c5255d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:30.280125 30558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a808961b4f44115b8f5ff5ae3295cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:30.280133 31009 tablet_service.cc:1467] Processing CreateTablet for tablet 9a808961b4f44115b8f5ff5ae3295cbf (DEFAULT_TABLE table=test-table [id=dc432ac6a2ca46d1bfb5a7bc9c5255d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:30.280658 31009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a808961b4f44115b8f5ff5ae3295cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:30.285660 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 9a808961b4f44115b8f5ff5ae3295cbf (DEFAULT_TABLE table=test-table [id=dc432ac6a2ca46d1bfb5a7bc9c5255d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:30.286183 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a808961b4f44115b8f5ff5ae3295cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:30.287951 31291 tablet_bootstrap.cc:492] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Bootstrap starting.
I20260807 06:48:30.290016 31291 tablet_bootstrap.cc:654] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:30.291970 31294 tablet_bootstrap.cc:492] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Bootstrap starting.
I20260807 06:48:30.292972 31297 tablet_bootstrap.cc:492] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Bootstrap starting.
I20260807 06:48:30.294034 31294 tablet_bootstrap.cc:654] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:30.294937 31297 tablet_bootstrap.cc:654] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:30.304628 31297 tablet_bootstrap.cc:492] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: No bootstrap required, opened a new log
I20260807 06:48:30.304724 31291 tablet_bootstrap.cc:492] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: No bootstrap required, opened a new log
I20260807 06:48:30.304873 31297 ts_tablet_manager.cc:1397] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260807 06:48:30.304999 31291 ts_tablet_manager.cc:1397] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260807 06:48:30.305842 31297 raft_consensus.cc:348] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.306005 31291 raft_consensus.cc:348] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.306190 31297 raft_consensus.cc:374] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:30.306290 31297 raft_consensus.cc:729] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f258fde344497b06afaf8db3846ff, State: Initialized, Role: FOLLOWER
I20260807 06:48:30.306340 31291 raft_consensus.cc:374] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:30.306460 31291 raft_consensus.cc:729] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Initialized, Role: FOLLOWER
I20260807 06:48:30.306598 31297 consensus_queue.cc:260] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.306963 31294 tablet_bootstrap.cc:492] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: No bootstrap required, opened a new log
I20260807 06:48:30.306825 31291 consensus_queue.cc:260] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.307158 31294 ts_tablet_manager.cc:1397] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260807 06:48:30.307452 31291 ts_tablet_manager.cc:1428] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260807 06:48:30.308130 31297 ts_tablet_manager.cc:1428] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:30.308081 31294 raft_consensus.cc:348] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.308367 31294 raft_consensus.cc:374] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:30.308461 31294 raft_consensus.cc:729] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3088a2d4967b44c7be0a90ce964ebd9c, State: Initialized, Role: FOLLOWER
I20260807 06:48:30.308734 31294 consensus_queue.cc:260] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.309303 31294 ts_tablet_manager.cc:1428] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 06:48:30.368180 31251 raft_consensus.cc:482] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:30.368412 31251 raft_consensus.cc:504] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.369254 31251 leader_election.cc:288] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:30.369984 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a808961b4f44115b8f5ff5ae3295cbf" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff" is_pre_election: true
I20260807 06:48:30.370031 31049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a808961b4f44115b8f5ff5ae3295cbf" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" is_pre_election: true
I20260807 06:48:30.370337 30812 raft_consensus.cc:2454] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 0.
I20260807 06:48:30.370404 31049 raft_consensus.cc:2454] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 0.
I20260807 06:48:30.370983 30499 leader_election.cc:302] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395f258fde344497b06afaf8db3846ff, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:30.371328 31251 raft_consensus.cc:2781] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:30.371491 31251 raft_consensus.cc:482] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:30.371608 31251 raft_consensus.cc:3037] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:30.375161 31251 raft_consensus.cc:504] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.375962 31251 leader_election.cc:288] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 election: Requested vote from peers 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155), 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011)
I20260807 06:48:30.376591 31049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a808961b4f44115b8f5ff5ae3295cbf" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3088a2d4967b44c7be0a90ce964ebd9c"
I20260807 06:48:30.376628 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a808961b4f44115b8f5ff5ae3295cbf" candidate_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f258fde344497b06afaf8db3846ff"
I20260807 06:48:30.376854 31049 raft_consensus.cc:3037] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:30.376920 30812 raft_consensus.cc:3037] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:30.379997 31049 raft_consensus.cc:2454] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 1.
I20260807 06:48:30.380218 30812 raft_consensus.cc:2454] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bffbb70cb1cd4b749ca5bbf43a7dffaf in term 1.
I20260807 06:48:30.380605 30500 leader_election.cc:302] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3088a2d4967b44c7be0a90ce964ebd9c, bffbb70cb1cd4b749ca5bbf43a7dffaf; no voters:
I20260807 06:48:30.381016 31251 raft_consensus.cc:2781] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:30.381608 31251 raft_consensus.cc:686] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 LEADER]: Becoming Leader. State: Replica: bffbb70cb1cd4b749ca5bbf43a7dffaf, State: Running, Role: LEADER
I20260807 06:48:30.382251 31251 consensus_queue.cc:237] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } }
I20260807 06:48:30.385841 30360 catalog_manager.cc:5495] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf reported cstate change: term changed from 0 to 1, leader changed from <none> to bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65). New cstate: current_term: 1 leader_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffbb70cb1cd4b749ca5bbf43a7dffaf" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 } health_report { overall_health: UNKNOWN } } }
I20260807 06:48:30.414165 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:30.407847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.27.54.65:44731)
0807 06:48:30.408643 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:30.408647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:30.408684 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:48:30.408746 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:30.408749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:30.408812 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:30.408930 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:30.409407 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.410539 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.412124 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:30.412600 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:30.412760 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:30.413292 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:30.413428 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:30.413534 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:30.413708 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 06:48:30.413796 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":433,"threads_started":1}
I20260807 06:48:30.425930 31049 raft_consensus.cc:1261] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Refusing update from remote peer bffbb70cb1cd4b749ca5bbf43a7dffaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:30.425997 30812 raft_consensus.cc:1261] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Refusing update from remote peer bffbb70cb1cd4b749ca5bbf43a7dffaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:30.426616 31322 consensus_queue.cc:1035] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f258fde344497b06afaf8db3846ff" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:30.427284 31322 consensus_queue.cc:1035] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3088a2d4967b44c7be0a90ce964ebd9c" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:48:30.450809 31259 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 06:48:32.048056 30360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:32.048290 30360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:32.052716 30360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e5ce310701542a6b9184d085fd7a91f: Sending DeleteTablet for 3 replicas of tablet 9a808961b4f44115b8f5ff5ae3295cbf
I20260807 06:48:32.053678 30558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a808961b4f44115b8f5ff5ae3295cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:32 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:35974
I20260807 06:48:32.053690 30780 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a808961b4f44115b8f5ff5ae3295cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:32 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60016
I20260807 06:48:32.053786 31006 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a808961b4f44115b8f5ff5ae3295cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:32 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47010
I20260807 06:48:32.055541 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30471
I20260807 06:48:32.067044 31369 tablet_replica.cc:330] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: stopping tablet replica
I20260807 06:48:32.067395 31370 tablet_replica.cc:330] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: stopping tablet replica
I20260807 06:48:32.067840 31370 raft_consensus.cc:2229] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:32.068161 31369 raft_consensus.cc:2229] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:32.068214 31370 raft_consensus.cc:2258] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:32.069521 31370 ts_tablet_manager.cc:1905] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:32.069842 31369 raft_consensus.cc:2258] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:32.071439 31369 ts_tablet_manager.cc:1905] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:32.073942 31367 tablet_replica.cc:330] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: stopping tablet replica
I20260807 06:48:32.074215 31367 raft_consensus.cc:2229] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:32.074522 31367 raft_consensus.cc:2258] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:32.076814 31370 ts_tablet_manager.cc:1918] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:48:32.077014 31370 log.cc:1196] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/9a808961b4f44115b8f5ff5ae3295cbf
I20260807 06:48:32.077492 31370 ts_tablet_manager.cc:1939] T 9a808961b4f44115b8f5ff5ae3295cbf P 3088a2d4967b44c7be0a90ce964ebd9c: Deleting consensus metadata
I20260807 06:48:32.078400 31367 ts_tablet_manager.cc:1905] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:32.078708 30338 catalog_manager.cc:4841] TS 3088a2d4967b44c7be0a90ce964ebd9c (127.27.54.67:37011): tablet 9a808961b4f44115b8f5ff5ae3295cbf (table test-table [id=dc432ac6a2ca46d1bfb5a7bc9c5255d7]) successfully deleted
I20260807 06:48:32.081190 31369 ts_tablet_manager.cc:1918] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:48:32.081334 31369 log.cc:1196] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/9a808961b4f44115b8f5ff5ae3295cbf
I20260807 06:48:32.081916 31369 ts_tablet_manager.cc:1939] T 9a808961b4f44115b8f5ff5ae3295cbf P bffbb70cb1cd4b749ca5bbf43a7dffaf: Deleting consensus metadata
I20260807 06:48:32.083316 30336 catalog_manager.cc:4841] TS bffbb70cb1cd4b749ca5bbf43a7dffaf (127.27.54.65:44731): tablet 9a808961b4f44115b8f5ff5ae3295cbf (table test-table [id=dc432ac6a2ca46d1bfb5a7bc9c5255d7]) successfully deleted
I20260807 06:48:32.084743 31367 ts_tablet_manager.cc:1918] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 06:48:32.084900 31367 log.cc:1196] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/9a808961b4f44115b8f5ff5ae3295cbf
I20260807 06:48:32.085414 31367 ts_tablet_manager.cc:1939] T 9a808961b4f44115b8f5ff5ae3295cbf P 395f258fde344497b06afaf8db3846ff: Deleting consensus metadata
I20260807 06:48:32.086488 30337 catalog_manager.cc:4841] TS 395f258fde344497b06afaf8db3846ff (127.27.54.66:38155): tablet 9a808961b4f44115b8f5ff5ae3295cbf (table test-table [id=dc432ac6a2ca46d1bfb5a7bc9c5255d7]) successfully deleted
Aug 07 06:48:32 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:32.096096 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:32.057306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.27.54.65:44731)
0807 06:48:32.072902 (+ 15596us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:32.072912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:32.073507 (+ 595us) server_negotiation.cc:407] Connection header received
0807 06:48:32.073798 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:32.073803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:32.073891 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:32.074009 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:32.076399 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:32.077548 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:32.080181 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:32.080703 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:32.080874 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:32.089381 (+ 8507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:32.089402 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:32.089419 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:32.089476 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:32.092667 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:32.093465 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:32.093474 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:32.093477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:32.093670 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:32.094111 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:32.094115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:32.094117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:32.094548 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:32.094908 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:32.095323 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 06:48:32.095595 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15383,"thread_start_us":16064,"threads_started":1}
W20260807 06:48:32.097059 30669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33372
W20260807 06:48:32.098083 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30471: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:32.098340 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 30471
I20260807 06:48:32.144747 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30682
Aug 07 06:48:32 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:32.162140 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:32.145587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.27.54.66:38155)
0807 06:48:32.146180 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:32.146184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:32.146201 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:48:32.146257 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:32.146260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:32.146319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:32.146447 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:32.147926 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:32.148728 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:32.150064 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:32.150477 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:32.150619 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:32.155179 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:32.155197 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:32.155214 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:32.155252 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:32.158617 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:32.159545 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:32.159563 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:32.159566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:32.159740 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:32.160151 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:32.160156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:32.160159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:32.160740 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:32.161097 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:32.161506 (+ 409us) server_negotiation.cc:299] Negotiation successful
0807 06:48:32.161764 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":321,"threads_started":1}
W20260807 06:48:32.162889 30888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54416
W20260807 06:48:32.163554 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30682: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:32.163762 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 30682
I20260807 06:48:32.198137 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30904
Aug 07 06:48:32 dist-test-slave-3kk6 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:32.215047 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:32.198981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.27.54.67:37011)
0807 06:48:32.199563 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:32.199568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:32.199585 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:48:32.199645 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:32.199649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:32.199716 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:32.199818 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:32.201165 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:32.201928 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:32.203400 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:32.203853 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:32.203953 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:32.208734 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:32.208753 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:32.208769 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:32.208813 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:32.212240 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:32.213029 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:32.213037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:32.213039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:32.213191 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:32.213581 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:32.213585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:32.213587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:32.213984 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:32.214214 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:32.214468 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 06:48:32.214729 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":326,"threads_started":1}
W20260807 06:48:32.215755 31114 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36082
W20260807 06:48:32.216442 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30904: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:32.216620 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 30904
I20260807 06:48:32.260699 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 30307
I20260807 06:48:32.262070 30426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:32.386641 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 30307
2026-08-07T06:48:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9156 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:32 dist-test-slave-3kk6 krb5kdc[31379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:32 dist-test-slave-3kk6 krb5kdc[31379](info): set up 2 sockets
Aug 07 06:48:32 dist-test-slave-3kk6 krb5kdc[31379](info): commencing operation
krb5kdc: starting...
W20260807 06:48:33.690932 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:33 dist-test-slave-3kk6 krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:33.947419 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34485
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:34.301967 31397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:34.302299 31397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:34.302395 31397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:34.309955 31397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:34.310092 31397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:34.310141 31397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:34.310197 31397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:34.310245 31397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:34.320369 31397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:34485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:34.321799 31397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:34.323207 31397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:34.336237 31405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:34.336335 31404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:34.336272 31407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:34.336606 31397 server_base.cc:1034] running on GCE node
I20260807 06:48:34.337769 31397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:34.339394 31397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:34.340649 31397 hybrid_clock.cc:648] HybridClock initialized: now 1786085314340562 us; error 74 us; skew 500 ppm
I20260807 06:48:34.341066 31397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:34 dist-test-slave-3kk6 krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:34.344554 31397 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:34.346213 31397 webserver.cc:460] Webserver started at http://127.27.54.126:40495/ using document root <none> and password file <none>
I20260807 06:48:34.347084 31397 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:34.347239 31397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:34.347572 31397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:34.350814 31397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "82f3841601f849c984376c925fb744bd"
format_stamp: "Formatted at 2026-08-07 06:48:34 on dist-test-slave-3kk6"
server_key: "bd69694ce889d75149035e1a4b4bd20c"
server_key_iv: "f88e4155b493566beea9aa68f12d8ed7"
server_key_version: "encryptionkey@0"
I20260807 06:48:34.351751 31397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "82f3841601f849c984376c925fb744bd"
format_stamp: "Formatted at 2026-08-07 06:48:34 on dist-test-slave-3kk6"
server_key: "bd69694ce889d75149035e1a4b4bd20c"
server_key_iv: "f88e4155b493566beea9aa68f12d8ed7"
server_key_version: "encryptionkey@0"
I20260807 06:48:34.357821 31397 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 06:48:34.361613 31418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:34.363086 31397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:34.363284 31397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "82f3841601f849c984376c925fb744bd"
format_stamp: "Formatted at 2026-08-07 06:48:34 on dist-test-slave-3kk6"
server_key: "bd69694ce889d75149035e1a4b4bd20c"
server_key_iv: "f88e4155b493566beea9aa68f12d8ed7"
server_key_version: "encryptionkey@0"
I20260807 06:48:34.363511 31397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:34.377027 31397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:34.381976 31397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:34.382310 31397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:34.404191 31397 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:34485
I20260807 06:48:34.404210 31504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:34485 every 8 connection(s)
I20260807 06:48:34.405854 31397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:34.409842 31505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:34.413245 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 31397
I20260807 06:48:34.413475 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:34.414155 27865 external_mini_cluster.cc:1428] Setting key 97434366c2a3fd7b632974306161f826
I20260807 06:48:34.423010 31505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd: Bootstrap starting.
I20260807 06:48:34.427058 31505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:34.428397 31505 log.cc:824] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:34.431859 31505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd: No bootstrap required, opened a new log
Aug 07 06:48:34 dist-test-slave-3kk6 krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:34.443518 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:34.416712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.27.54.126:34485)
0807 06:48:34.417756 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:34.417772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:34.418324 (+ 552us) server_negotiation.cc:407] Connection header received
0807 06:48:34.421074 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:34.421108 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:34.421680 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:34.422159 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:34.423931 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:34.426029 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:34.427580 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:34.428672 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:34.428866 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:34.433367 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:34.433379 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:34.433404 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:34.433462 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:34.438125 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:34.439181 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:34.439191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:34.439201 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:34.439443 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:34.439874 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:34.439879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:34.439883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:34.440705 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:34.441048 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:34.441468 (+ 420us) server_negotiation.cc:299] Negotiation successful
0807 06:48:34.441964 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":467,"threads_started":1}
I20260807 06:48:34.443171 31505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f3841601f849c984376c925fb744bd" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:48:34.443753 31505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:34.443907 31505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f3841601f849c984376c925fb744bd, State: Initialized, Role: FOLLOWER
I20260807 06:48:34.444551 31505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f3841601f849c984376c925fb744bd" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:48:34.444823 31505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:34.444967 31505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:34.445163 31505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:34.448148 31505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f3841601f849c984376c925fb744bd" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:48:34.448623 31505 leader_election.cc:302] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82f3841601f849c984376c925fb744bd; no voters:
I20260807 06:48:34.449213 31505 leader_election.cc:288] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:34.449365 31512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:34.450104 31512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [term 1 LEADER]: Becoming Leader. State: Replica: 82f3841601f849c984376c925fb744bd, State: Running, Role: LEADER
I20260807 06:48:34.450614 31512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f3841601f849c984376c925fb744bd" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:48:34.451501 31505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:34.453526 31516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f3841601f849c984376c925fb744bd. Latest consensus state: current_term: 1 leader_uuid: "82f3841601f849c984376c925fb744bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f3841601f849c984376c925fb744bd" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } } }
I20260807 06:48:34.453907 31516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:34.458920 31513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82f3841601f849c984376c925fb744bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f3841601f849c984376c925fb744bd" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } } }
I20260807 06:48:34.459720 31513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd [sys.catalog]: This master's current role is: LEADER
W20260807 06:48:34.463963 31538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:48:34.464109 31538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:48:34.464270 31522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:34.466519 31522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:34.475371 31522 catalog_manager.cc:1347] Generated new cluster ID: 5f525f01c0eb4b67a025b0f847b28e1f
I20260807 06:48:34.475517 31522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:34.490546 31522 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:34.492527 31522 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:34.501499 31522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82f3841601f849c984376c925fb744bd: Generated new TSK 0
I20260807 06:48:34.502641 31522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:34.564567 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34485
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:34.928998 31551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:34.929317 31551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:34.929505 31551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:34.937150 31551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:34.937294 31551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:34.937506 31551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:34.948238 31551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:34.949762 31551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:34.951364 31551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:34.966727 31556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:34.966751 31559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:34.966609 31557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:34.967463 31551 server_base.cc:1034] running on GCE node
I20260807 06:48:34.968248 31551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:34.969386 31551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:34.970690 31551 hybrid_clock.cc:648] HybridClock initialized: now 1786085314970612 us; error 36 us; skew 500 ppm
I20260807 06:48:34.971179 31551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:34 dist-test-slave-3kk6 krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:34.975790 31551 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:34.978101 31551 webserver.cc:460] Webserver started at http://127.27.54.65:46617/ using document root <none> and password file <none>
I20260807 06:48:34.979440 31551 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:34.979642 31551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:34.980124 31551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:34.984338 31551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "2ed8d7a820b5480e8cc09b4022b1f94c"
format_stamp: "Formatted at 2026-08-07 06:48:34 on dist-test-slave-3kk6"
server_key: "cabeb610deaace74baa89633a8ecc8a7"
server_key_iv: "e37d86d668423b5d1a3eba549d6238cb"
server_key_version: "encryptionkey@0"
I20260807 06:48:34.985342 31551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "2ed8d7a820b5480e8cc09b4022b1f94c"
format_stamp: "Formatted at 2026-08-07 06:48:34 on dist-test-slave-3kk6"
server_key: "cabeb610deaace74baa89633a8ecc8a7"
server_key_iv: "e37d86d668423b5d1a3eba549d6238cb"
server_key_version: "encryptionkey@0"
I20260807 06:48:34.991429 31551 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 06:48:34.995425 31567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:34.997036 31551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:34.997267 31551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "2ed8d7a820b5480e8cc09b4022b1f94c"
format_stamp: "Formatted at 2026-08-07 06:48:34 on dist-test-slave-3kk6"
server_key: "cabeb610deaace74baa89633a8ecc8a7"
server_key_iv: "e37d86d668423b5d1a3eba549d6238cb"
server_key_version: "encryptionkey@0"
I20260807 06:48:34.997521 31551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:35.028918 31551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:35.034004 31551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:35.034385 31551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:35.035524 31551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:35.036993 31551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:35.037101 31551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:35.037261 31551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:35.037349 31551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:35.079356 31551 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43129
I20260807 06:48:35.079468 31737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43129 every 8 connection(s)
I20260807 06:48:35.080765 31551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:35.083010 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 31551
I20260807 06:48:35.083235 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:35.083905 27865 external_mini_cluster.cc:1428] Setting key e0949c3af480e45e9082bc1982c6e28d
Aug 07 06:48:35 dist-test-slave-3kk6 krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:35.105209 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:35.083964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51961 (local address 127.27.54.126:34485)
0807 06:48:35.084513 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:35.084519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:35.085874 (+ 1355us) server_negotiation.cc:407] Connection header received
0807 06:48:35.087877 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:35.087884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:35.087984 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:35.088110 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:35.089663 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.090411 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.092089 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.092502 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.092646 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:35.096479 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:35.096505 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:35.096513 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:35.096564 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:35.100642 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:35.101840 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:35.101849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:35.101855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:35.102020 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:35.102626 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:35.102634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:35.102641 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:35.103107 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:35.103291 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:35.104402 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0807 06:48:35.104679 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
I20260807 06:48:35.106637 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:35.084577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34485 (local address 127.27.54.65:51961)
0807 06:48:35.085701 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:35.085742 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:35.087582 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:35.088376 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:35.088385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:35.088908 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:35.089480 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:35.089498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.090577 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.090590 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:35.091925 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:35.091938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.092757 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.092764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:35.093014 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:35.093758 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:35.093782 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:35.096222 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:35.100857 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:35.100869 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:35.100890 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:35.101662 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:35.102153 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:35.102162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:35.102167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:35.102459 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:35.103269 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:35.103279 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:35.103804 (+ 525us) client_negotiation.cc:769] Sending connection context
0807 06:48:35.104288 (+ 484us) client_negotiation.cc:240] Negotiation successful
0807 06:48:35.104973 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":612,"threads_started":1}
I20260807 06:48:35.108903 31740 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34485
I20260807 06:48:35.109284 31740 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:35.110284 31740 heartbeater.cc:511] Master 127.27.54.126:34485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
I20260807 06:48:35.113055 31446 ts_manager.cc:194] Registered new tserver with Master: 2ed8d7a820b5480e8cc09b4022b1f94c (127.27.54.65:43129)
I20260807 06:48:35.114573 31446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:51961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:35.154959 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34485
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:35.507846 31749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:35.508169 31749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:35.508332 31749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:35.516160 31749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:35.516312 31749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:35.516548 31749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:35.526929 31749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:35.528461 31749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:35.529861 31749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:35.544729 31755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:35.544809 31756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:35.545068 31758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:35.545651 31749 server_base.cc:1034] running on GCE node
I20260807 06:48:35.546384 31749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:35.547571 31749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:35.548807 31749 hybrid_clock.cc:648] HybridClock initialized: now 1786085315548764 us; error 45 us; skew 500 ppm
I20260807 06:48:35.549165 31749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:35 dist-test-slave-3kk6 krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:35.553025 31749 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:35.554816 31749 webserver.cc:460] Webserver started at http://127.27.54.66:34987/ using document root <none> and password file <none>
I20260807 06:48:35.555555 31749 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:35.555670 31749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:35.556080 31749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:35.559672 31749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "2e564db915c149479e009b2eaea38309"
format_stamp: "Formatted at 2026-08-07 06:48:35 on dist-test-slave-3kk6"
server_key: "e7b02dd5cc0c0b2710db7cfe89c01726"
server_key_iv: "3f3eb30c5712bddb6e4ba8d092a26e33"
server_key_version: "encryptionkey@0"
I20260807 06:48:35.560601 31749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "2e564db915c149479e009b2eaea38309"
format_stamp: "Formatted at 2026-08-07 06:48:35 on dist-test-slave-3kk6"
server_key: "e7b02dd5cc0c0b2710db7cfe89c01726"
server_key_iv: "3f3eb30c5712bddb6e4ba8d092a26e33"
server_key_version: "encryptionkey@0"
I20260807 06:48:35.567154 31749 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 06:48:35.571383 31772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:35.572881 31749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:48:35.573094 31749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "2e564db915c149479e009b2eaea38309"
format_stamp: "Formatted at 2026-08-07 06:48:35 on dist-test-slave-3kk6"
server_key: "e7b02dd5cc0c0b2710db7cfe89c01726"
server_key_iv: "3f3eb30c5712bddb6e4ba8d092a26e33"
server_key_version: "encryptionkey@0"
I20260807 06:48:35.573343 31749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:35.603890 31749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:35.609122 31749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:35.609463 31749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:35.610532 31749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:35.612130 31749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:35.612242 31749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:35.612393 31749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:35.612468 31749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:35.648499 31749 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:46137
I20260807 06:48:35.648558 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:46137 every 8 connection(s)
I20260807 06:48:35.649899 31749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:35.652995 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 31749
I20260807 06:48:35.653231 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:35.653807 27865 external_mini_cluster.cc:1428] Setting key cd9a07ffe626210d3af156d4a3ea3d0c
Aug 07 06:48:35 dist-test-slave-3kk6 krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:35.676275 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:35.652846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:33219 (local address 127.27.54.126:34485)
0807 06:48:35.653502 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:35.653507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:35.657296 (+ 3789us) server_negotiation.cc:407] Connection header received
0807 06:48:35.657388 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:35.657398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:35.657471 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:35.657562 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:35.660026 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.660755 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.663427 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.663874 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.663980 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:35.668312 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:35.668322 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:35.668331 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:35.668382 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:35.672293 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:35.673222 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:35.673231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:35.673237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:35.673402 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:35.673952 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:35.673960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:35.673966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:35.674422 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:35.674591 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:35.675454 (+ 863us) server_negotiation.cc:299] Negotiation successful
0807 06:48:35.675740 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":404,"threads_started":1}
I20260807 06:48:35.676972 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:35.653671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34485 (local address 127.27.54.66:33219)
0807 06:48:35.654544 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:35.654601 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:35.656928 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:35.657907 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:35.657924 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:35.658899 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:35.659800 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:35.659832 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.660931 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.660940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:35.663229 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:35.663256 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:35.664125 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:35.664135 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:35.664480 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:35.665378 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:35.665415 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:35.668022 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:35.672515 (+ 4493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:35.672524 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:35.672539 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:35.673052 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:35.673557 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:35.673561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:35.673564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:35.673795 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:35.674557 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:35.674563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:35.675034 (+ 471us) client_negotiation.cc:769] Sending connection context
0807 06:48:35.675295 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 06:48:35.675733 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":336,"threads_started":1}
I20260807 06:48:35.678890 31943 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34485
I20260807 06:48:35.679226 31943 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:35.680014 31943 heartbeater.cc:511] Master 127.27.54.126:34485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
I20260807 06:48:35.681671 31446 ts_manager.cc:194] Registered new tserver with Master: 2e564db915c149479e009b2eaea38309 (127.27.54.66:46137)
I20260807 06:48:35.682586 31446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:33219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:35.714553 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34485
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:36.065390 31954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:36.065696 31954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:36.065866 31954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:36.073390 31954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:36.073524 31954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:36.073706 31954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:36.084050 31954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:36.085355 31954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:36.086782 31954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:36.103466 31963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:36.103480 31965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:36.104584 31954 server_base.cc:1034] running on GCE node
W20260807 06:48:36.104857 31961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:36.105710 31954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:36.106950 31954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:36.108206 31954 hybrid_clock.cc:648] HybridClock initialized: now 1786085316108149 us; error 40 us; skew 500 ppm
I20260807 06:48:36.108592 31954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:36 dist-test-slave-3kk6 krb5kdc[31379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:36.111959 31954 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:36.113659 31954 webserver.cc:460] Webserver started at http://127.27.54.67:45365/ using document root <none> and password file <none>
I20260807 06:48:36.114421 31954 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:36.114580 31954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:36.114948 31954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:36.117997 31954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "f2ab47c1cbcd48b4b389532c3fa99795"
format_stamp: "Formatted at 2026-08-07 06:48:36 on dist-test-slave-3kk6"
server_key: "6dd61fc79f44363d0fb4d0b53d2e7570"
server_key_iv: "02dc63dab9f887ae13dc317eb30a9e4b"
server_key_version: "encryptionkey@0"
I20260807 06:48:36.118639 31740 heartbeater.cc:503] Master 127.27.54.126:34485 was elected leader, sending a full tablet report...
I20260807 06:48:36.118992 31954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "f2ab47c1cbcd48b4b389532c3fa99795"
format_stamp: "Formatted at 2026-08-07 06:48:36 on dist-test-slave-3kk6"
server_key: "6dd61fc79f44363d0fb4d0b53d2e7570"
server_key_iv: "02dc63dab9f887ae13dc317eb30a9e4b"
server_key_version: "encryptionkey@0"
I20260807 06:48:36.124806 31954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 06:48:36.128449 31978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:36.129820 31954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:48:36.129992 31954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "f2ab47c1cbcd48b4b389532c3fa99795"
format_stamp: "Formatted at 2026-08-07 06:48:36 on dist-test-slave-3kk6"
server_key: "6dd61fc79f44363d0fb4d0b53d2e7570"
server_key_iv: "02dc63dab9f887ae13dc317eb30a9e4b"
server_key_version: "encryptionkey@0"
I20260807 06:48:36.130206 31954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:36.153486 31954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:36.157917 31954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:36.158178 31954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:36.159240 31954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:36.160801 31954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:36.160907 31954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:36.161067 31954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:36.161140 31954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:36.196005 31954 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:45973
I20260807 06:48:36.196097 32135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:45973 every 8 connection(s)
I20260807 06:48:36.197358 31954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:36.200400 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 31954
I20260807 06:48:36.200605 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:36.201207 27865 external_mini_cluster.cc:1428] Setting key 47fc35edb56e1c17259efa9f17045f5a
Aug 07 06:48:36 dist-test-slave-3kk6 krb5kdc[31379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:36.219892 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:36.200295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:43875 (local address 127.27.54.126:34485)
0807 06:48:36.200915 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:36.200922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:36.201864 (+ 942us) server_negotiation.cc:407] Connection header received
0807 06:48:36.204293 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:36.204300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:36.204409 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:36.204530 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:36.206121 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:36.207224 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:36.208808 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:36.209207 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:36.209307 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:36.213067 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:36.213073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:36.213076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:36.213109 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:36.216438 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:36.217242 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:36.217249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:36.217253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:36.217391 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:36.217853 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:36.217861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:36.217866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:36.218152 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:36.218323 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:36.219323 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0807 06:48:36.219573 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":319,"threads_started":1}
I20260807 06:48:36.220963 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:36.200831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34485 (local address 127.27.54.67:43875)
0807 06:48:36.201660 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:36.201718 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:36.203952 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:36.204824 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:36.204833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:36.205383 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:36.205967 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:36.205985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:36.207371 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:36.207376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:36.208660 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:36.208683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:36.209443 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:36.209451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:36.209690 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:36.210460 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:36.210483 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:36.212835 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:36.216595 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:36.216604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:36.216624 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:36.217097 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:36.217513 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:36.217519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:36.217522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:36.217733 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:36.218292 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:36.218300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:36.218823 (+ 523us) client_negotiation.cc:769] Sending connection context
0807 06:48:36.219189 (+ 366us) client_negotiation.cc:240] Negotiation successful
0807 06:48:36.219784 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":352,"threads_started":1}
I20260807 06:48:36.222421 32136 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34485
I20260807 06:48:36.222772 32136 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:36.223469 32136 heartbeater.cc:511] Master 127.27.54.126:34485 requested a full tablet report, sending...
I20260807 06:48:36.224951 31446 ts_manager.cc:194] Registered new tserver with Master: f2ab47c1cbcd48b4b389532c3fa99795 (127.27.54.67:45973)
I20260807 06:48:36.225737 31446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:43875
I20260807 06:48:36.229570 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 06:48:36.270763 32138 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:48:36.267866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.27.54.126:34485)
0807 06:48:36.268059 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:36.268063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:36.268640 (+ 577us) server_negotiation.cc:407] Connection header received
0807 06:48:36.268847 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:36.268852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:36.268935 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:36.269036 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:36.270491 (+ 1455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56822: BlockingRecv error: recv got EOF from 127.0.0.1:56822 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:48:36.275322 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31551
W20260807 06:48:36.278338 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31551: Not authorized: Client connection negotiation failed: client connection to 127.27.54.65:43129: server requires authentication, but client does not have Kerberos credentials available
I20260807 06:48:36.278575 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 31551
I20260807 06:48:36.303252 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31749
W20260807 06:48:36.306226 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31749: Not authorized: Client connection negotiation failed: client connection to 127.27.54.66:46137: server requires authentication, but client does not have Kerberos credentials available
I20260807 06:48:36.306406 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 31749
I20260807 06:48:36.327021 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31954
W20260807 06:48:36.330524 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31954: Not authorized: Client connection negotiation failed: client connection to 127.27.54.67:45973: server requires authentication, but client does not have Kerberos credentials available
I20260807 06:48:36.330739 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 31954
I20260807 06:48:36.355680 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 31397
I20260807 06:48:36.357153 31496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:36.466529 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 31397
2026-08-07T06:48:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4087 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:36 dist-test-slave-3kk6 krb5kdc[32163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:36 dist-test-slave-3kk6 krb5kdc[32163](info): set up 2 sockets
Aug 07 06:48:36 dist-test-slave-3kk6 krb5kdc[32163](info): commencing operation
krb5kdc: starting...
W20260807 06:48:37.774425 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:37 dist-test-slave-3kk6 krb5kdc[32163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:38.022071 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:33023
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:38.367302 32181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:38.367681 32181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:38.367861 32181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:38.375141 32181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:38.375298 32181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:38.375411 32181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:38.375519 32181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:38.375614 32181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:38.385458 32181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:38.386831 32181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:38.388193 32181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:38.401217 32188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:38.401154 32190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:38.401254 32197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:38.402421 32181 server_base.cc:1034] running on GCE node
I20260807 06:48:38.403317 32181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:38.404664 32181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:38.405908 32181 hybrid_clock.cc:648] HybridClock initialized: now 1786085318405873 us; error 48 us; skew 500 ppm
I20260807 06:48:38.406256 32181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:38 dist-test-slave-3kk6 krb5kdc[32163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:38.414595 32181 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:38.416150 32181 webserver.cc:460] Webserver started at http://127.27.54.126:36065/ using document root <none> and password file <none>
I20260807 06:48:38.416836 32181 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:38.416997 32181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:38.417322 32181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:38.420431 32181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "4510f711812248a3855c23e78182ff97"
format_stamp: "Formatted at 2026-08-07 06:48:38 on dist-test-slave-3kk6"
server_key: "18760d83fa8a3e656668ca5ad629f6df"
server_key_iv: "11a294dd58230f2b836a98bb53c259e7"
server_key_version: "encryptionkey@0"
I20260807 06:48:38.421277 32181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "4510f711812248a3855c23e78182ff97"
format_stamp: "Formatted at 2026-08-07 06:48:38 on dist-test-slave-3kk6"
server_key: "18760d83fa8a3e656668ca5ad629f6df"
server_key_iv: "11a294dd58230f2b836a98bb53c259e7"
server_key_version: "encryptionkey@0"
I20260807 06:48:38.426335 32181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:48:38.429637 32210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:38.431016 32181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:48:38.431191 32181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "4510f711812248a3855c23e78182ff97"
format_stamp: "Formatted at 2026-08-07 06:48:38 on dist-test-slave-3kk6"
server_key: "18760d83fa8a3e656668ca5ad629f6df"
server_key_iv: "11a294dd58230f2b836a98bb53c259e7"
server_key_version: "encryptionkey@0"
I20260807 06:48:38.431391 32181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:38.462538 32181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:38.467496 32181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:38.467854 32181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:38.486580 32181 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:33023
I20260807 06:48:38.486614 32287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:33023 every 8 connection(s)
I20260807 06:48:38.488078 32181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:38.492009 32288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:38.496958 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 32181
I20260807 06:48:38.497241 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:38.498090 27865 external_mini_cluster.cc:1428] Setting key 325c27a9d0a0144f4c42e070fc03dcf5
I20260807 06:48:38.504273 32288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97: Bootstrap starting.
I20260807 06:48:38.508087 32288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:38.509325 32288 log.cc:824] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:38.512864 32288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97: No bootstrap required, opened a new log
Aug 07 06:48:38 dist-test-slave-3kk6 krb5kdc[32163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:38.520797 32288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4510f711812248a3855c23e78182ff97" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 33023 } }
I20260807 06:48:38.521224 32288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:38.521358 32288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4510f711812248a3855c23e78182ff97, State: Initialized, Role: FOLLOWER
I20260807 06:48:38.522020 32288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4510f711812248a3855c23e78182ff97" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 33023 } }
I20260807 06:48:38.522272 32288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:38.522423 32288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:38.522620 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:38.500166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.27.54.126:33023)
0807 06:48:38.501049 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:38.501061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:38.501102 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:48:38.502981 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:38.503005 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:38.503392 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:38.503722 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:38.505662 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:38.507081 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:38.508901 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:38.509627 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:38.509765 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:38.514795 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:38.514803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:38.514820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:38.514861 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:38.518158 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:38.518963 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:38.518976 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:38.518982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:38.519157 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:38.519569 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:38.519577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:38.519587 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:38.520160 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:38.520476 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:38.520940 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 06:48:38.521432 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":422,"threads_started":1}
I20260807 06:48:38.522622 32288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:38.525825 32288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4510f711812248a3855c23e78182ff97" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 33023 } }
I20260807 06:48:38.526444 32288 leader_election.cc:302] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4510f711812248a3855c23e78182ff97; no voters:
I20260807 06:48:38.526939 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:48:38.527186 32288 leader_election.cc:288] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:38.527309 32294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:38.529147 32288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:38.535589 32294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [term 1 LEADER]: Becoming Leader. State: Replica: 4510f711812248a3855c23e78182ff97, State: Running, Role: LEADER
I20260807 06:48:38.536362 32294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4510f711812248a3855c23e78182ff97" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 33023 } }
I20260807 06:48:38.539862 32296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4510f711812248a3855c23e78182ff97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4510f711812248a3855c23e78182ff97" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 33023 } } }
I20260807 06:48:38.540365 32296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:38.540822 32297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4510f711812248a3855c23e78182ff97. Latest consensus state: current_term: 1 leader_uuid: "4510f711812248a3855c23e78182ff97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4510f711812248a3855c23e78182ff97" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 33023 } } }
I20260807 06:48:38.541232 32297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:38.541581 32312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:38.547152 32312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:38.567559 32312 catalog_manager.cc:1347] Generated new cluster ID: fb30abc407b6442fb52c9879b706e24b
I20260807 06:48:38.567687 32312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:38.584870 32312 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:38.586815 32312 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:38.590735 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:38.580901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.27.54.126:33023)
0807 06:48:38.581123 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:38.581133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:38.581767 (+ 634us) server_negotiation.cc:407] Connection header received
0807 06:48:38.582011 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:38.582016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:38.582085 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:38.582201 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:38.583573 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:38.584464 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:38.586425 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:38.587202 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:38.587353 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:38.588740 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:38.588747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:38.588752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:38.589788 (+ 1036us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:38.589876 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:38.589906 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:38.590035 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:38.590180 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 06:48:38.590320 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:48:38.601260 32312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4510f711812248a3855c23e78182ff97: Generated new TSK 0
I20260807 06:48:38.602394 32312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:48:38.613147 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 32181
I20260807 06:48:38.614722 32282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:38.706012 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 32181
2026-08-07T06:48:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2222 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:38 dist-test-slave-3kk6 krb5kdc[32331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:38 dist-test-slave-3kk6 krb5kdc[32331](info): set up 2 sockets
Aug 07 06:48:38 dist-test-slave-3kk6 krb5kdc[32331](info): commencing operation
krb5kdc: starting...
W20260807 06:48:40.017783 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:40 dist-test-slave-3kk6 krb5kdc[32331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:40.282799 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:44955
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:40.634287 32349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:40.634619 32349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:40.634882 32349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:40.642333 32349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:40.642453 32349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:40.642563 32349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:40.642680 32349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:40.642751 32349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:40.653126 32349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:44955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:40.654565 32349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:40.655968 32349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:40.668366 32361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:40.668422 32359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:40.669008 32357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:40.669415 32349 server_base.cc:1034] running on GCE node
I20260807 06:48:40.670220 32349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:40.671836 32349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:40.673085 32349 hybrid_clock.cc:648] HybridClock initialized: now 1786085320673043 us; error 62 us; skew 500 ppm
I20260807 06:48:40.673455 32349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:40 dist-test-slave-3kk6 krb5kdc[32331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085320, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:40.676903 32349 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:40.678726 32349 webserver.cc:460] Webserver started at http://127.27.54.126:36255/ using document root <none> and password file <none>
I20260807 06:48:40.679418 32349 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:40.679525 32349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:40.679796 32349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:40.682984 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "d72906b03b734e159a419207a0a7c8a5"
format_stamp: "Formatted at 2026-08-07 06:48:40 on dist-test-slave-3kk6"
server_key: "de7b8f143e673c58769df5b7fe83b8c2"
server_key_iv: "dd66a6376de1971e3e8bbf9df8277e93"
server_key_version: "encryptionkey@0"
I20260807 06:48:40.683758 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "d72906b03b734e159a419207a0a7c8a5"
format_stamp: "Formatted at 2026-08-07 06:48:40 on dist-test-slave-3kk6"
server_key: "de7b8f143e673c58769df5b7fe83b8c2"
server_key_iv: "dd66a6376de1971e3e8bbf9df8277e93"
server_key_version: "encryptionkey@0"
I20260807 06:48:40.689216 32349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 06:48:40.692727 32372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:40.694113 32349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:48:40.694309 32349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "d72906b03b734e159a419207a0a7c8a5"
format_stamp: "Formatted at 2026-08-07 06:48:40 on dist-test-slave-3kk6"
server_key: "de7b8f143e673c58769df5b7fe83b8c2"
server_key_iv: "dd66a6376de1971e3e8bbf9df8277e93"
server_key_version: "encryptionkey@0"
I20260807 06:48:40.694489 32349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:40.732492 32349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:40.737440 32349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:40.737774 32349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:40.757978 32349 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:44955
I20260807 06:48:40.758011 32448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:44955 every 8 connection(s)
I20260807 06:48:40.759538 32349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:40.759960 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 32349
I20260807 06:48:40.760200 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:40.761102 27865 external_mini_cluster.cc:1428] Setting key f451a53e144d16725cb7df9dd4a992e8
I20260807 06:48:40.764114 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:48:40 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:40.775600 32449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5: Bootstrap starting.
I20260807 06:48:40.779331 32449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:40.780486 32449 log.cc:824] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:40.784144 32449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5: No bootstrap required, opened a new log
I20260807 06:48:40.784731 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:40.763386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.27.54.126:44955)
0807 06:48:40.764255 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:40.764273 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:40.764333 (+ 60us) server_negotiation.cc:407] Connection header received
0807 06:48:40.766001 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:40.766026 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:40.766418 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:40.766919 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:40.768429 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:40.769867 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:40.771212 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:40.771917 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:40.772175 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:40.776578 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:40.776586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:40.776602 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:40.776642 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:40.780041 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:40.780893 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:40.780900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:40.780907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:40.781227 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:40.781714 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:40.781720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:40.781724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:40.782400 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:40.782709 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:40.783211 (+ 502us) server_negotiation.cc:299] Negotiation successful
0807 06:48:40.783614 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
I20260807 06:48:40.792444 32449 raft_consensus.cc:348] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72906b03b734e159a419207a0a7c8a5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44955 } }
I20260807 06:48:40.792881 32449 raft_consensus.cc:374] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:40.793009 32449 raft_consensus.cc:729] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72906b03b734e159a419207a0a7c8a5, State: Initialized, Role: FOLLOWER
I20260807 06:48:40.793571 32449 consensus_queue.cc:260] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72906b03b734e159a419207a0a7c8a5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44955 } }
I20260807 06:48:40.793820 32449 raft_consensus.cc:388] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:40.793974 32449 raft_consensus.cc:482] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:40.794155 32449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:40.796545 32449 raft_consensus.cc:504] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72906b03b734e159a419207a0a7c8a5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44955 } }
I20260807 06:48:40.797083 32449 leader_election.cc:302] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d72906b03b734e159a419207a0a7c8a5; no voters:
I20260807 06:48:40.797804 32449 leader_election.cc:288] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:40.798012 32454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:40.798861 32454 raft_consensus.cc:686] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [term 1 LEADER]: Becoming Leader. State: Replica: d72906b03b734e159a419207a0a7c8a5, State: Running, Role: LEADER
I20260807 06:48:40.799368 32454 consensus_queue.cc:237] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72906b03b734e159a419207a0a7c8a5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44955 } }
I20260807 06:48:40.800089 32449 sys_catalog.cc:564] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:40.802685 32456 sys_catalog.cc:455] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d72906b03b734e159a419207a0a7c8a5. Latest consensus state: current_term: 1 leader_uuid: "d72906b03b734e159a419207a0a7c8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72906b03b734e159a419207a0a7c8a5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44955 } } }
I20260807 06:48:40.803133 32456 sys_catalog.cc:458] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:40.804237 32455 sys_catalog.cc:455] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d72906b03b734e159a419207a0a7c8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d72906b03b734e159a419207a0a7c8a5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44955 } } }
I20260807 06:48:40.804611 32455 sys_catalog.cc:458] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:40.805097 32466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:40.810380 32466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:40.819202 32466 catalog_manager.cc:1347] Generated new cluster ID: 2d93ee925a28426c8859b005a70429ef
I20260807 06:48:40.819366 32466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:40.893265 32466 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:40.894726 32466 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:40.917245 32466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d72906b03b734e159a419207a0a7c8a5: Generated new TSK 0
I20260807 06:48:40.918390 32466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:40.994762 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:44955
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:41.353291 32482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:41.353600 32482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:41.353773 32482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:41.361770 32482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:41.361912 32482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:41.362112 32482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:41.372398 32482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:41.373828 32482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:41.375219 32482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:41.388811 32492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:41.388947 32488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:41.388897 32490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:41.390048 32482 server_base.cc:1034] running on GCE node
I20260807 06:48:41.390764 32482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:41.391791 32482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:41.393023 32482 hybrid_clock.cc:648] HybridClock initialized: now 1786085321392957 us; error 63 us; skew 500 ppm
I20260807 06:48:41.393350 32482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:41 dist-test-slave-3kk6 krb5kdc[32331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:41.396764 32482 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:41.398278 32482 webserver.cc:460] Webserver started at http://127.27.54.65:40899/ using document root <none> and password file <none>
I20260807 06:48:41.399088 32482 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:41.399245 32482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:41.399575 32482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:41.402889 32482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "0fc8f7418c4f404a8f9c3ea7adb28581"
format_stamp: "Formatted at 2026-08-07 06:48:41 on dist-test-slave-3kk6"
server_key: "90f34f4400c1468e2effd064323fe86b"
server_key_iv: "6e4f3bf6eda9bf5e0fef926e36462af4"
server_key_version: "encryptionkey@0"
I20260807 06:48:41.403828 32482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "0fc8f7418c4f404a8f9c3ea7adb28581"
format_stamp: "Formatted at 2026-08-07 06:48:41 on dist-test-slave-3kk6"
server_key: "90f34f4400c1468e2effd064323fe86b"
server_key_iv: "6e4f3bf6eda9bf5e0fef926e36462af4"
server_key_version: "encryptionkey@0"
I20260807 06:48:41.409258 32482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 06:48:41.412964 32501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:41.414371 32482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:48:41.414558 32482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "0fc8f7418c4f404a8f9c3ea7adb28581"
format_stamp: "Formatted at 2026-08-07 06:48:41 on dist-test-slave-3kk6"
server_key: "90f34f4400c1468e2effd064323fe86b"
server_key_iv: "6e4f3bf6eda9bf5e0fef926e36462af4"
server_key_version: "encryptionkey@0"
I20260807 06:48:41.414800 32482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:41.438301 32482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:41.443120 32482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:41.443480 32482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:41.444543 32482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:41.445981 32482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:41.446089 32482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:41.446249 32482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:41.446326 32482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:41.482312 32482 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:45151
I20260807 06:48:41.482313 32678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:45151 every 8 connection(s)
I20260807 06:48:41.484052 32482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:41.492056 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 32482
I20260807 06:48:41.492306 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:41.492964 27865 external_mini_cluster.cc:1428] Setting key bad9656e2aeb6ca404d5fa4e1815c241
Aug 07 06:48:41 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:41.506554 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:41.487075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:40725 (local address 127.27.54.126:44955)
0807 06:48:41.487593 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:41.487600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:41.488668 (+ 1068us) server_negotiation.cc:407] Connection header received
0807 06:48:41.490518 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:41.490522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:41.490593 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:41.490767 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:41.492364 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:41.493197 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:41.495065 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:41.495492 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:41.495651 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:41.499769 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:41.499780 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:41.499784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:41.499819 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:41.502967 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:41.503855 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:41.503868 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:41.503904 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:41.504043 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:41.504577 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:41.504584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:41.504587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:41.504945 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:41.505105 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:41.506088 (+ 983us) server_negotiation.cc:299] Negotiation successful
0807 06:48:41.506271 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260807 06:48:41.508255 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:41.487734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:44955 (local address 127.27.54.65:40725)
0807 06:48:41.488500 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:41.488541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:41.490250 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:41.490952 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:41.490959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:41.491511 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:41.492117 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:41.492165 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:41.493375 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:41.493379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:41.494911 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:41.494924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:41.495736 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:41.495743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:41.496008 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:41.496784 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:41.496809 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:41.499484 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:41.503136 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:41.503146 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:41.503164 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:41.503721 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:41.504165 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:41.504170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:41.504174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:41.504460 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:41.505124 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:41.505133 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:41.505639 (+ 506us) client_negotiation.cc:769] Sending connection context
0807 06:48:41.506008 (+ 369us) client_negotiation.cc:240] Negotiation successful
0807 06:48:41.506676 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":288,"threads_started":1}
I20260807 06:48:41.510017 32680 heartbeater.cc:347] Connected to a master server at 127.27.54.126:44955
I20260807 06:48:41.510351 32680 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:41.511204 32680 heartbeater.cc:511] Master 127.27.54.126:44955 requested a full tablet report, sending...
I20260807 06:48:41.513425 32398 ts_manager.cc:194] Registered new tserver with Master: 0fc8f7418c4f404a8f9c3ea7adb28581 (127.27.54.65:45151)
I20260807 06:48:41.515004 32398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:40725
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:41.553830 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:44955
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:41.902696 32687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:41.903071 32687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:41.903298 32687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:41.911274 32687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:41.911414 32687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:41.911535 32687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:41.922322 32687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:41.923849 32687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:41.925230 32687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:41.939837 32694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:41.939878 32700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:41.939859 32698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:41.941185 32687 server_base.cc:1034] running on GCE node
I20260807 06:48:41.941891 32687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:41.943009 32687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:41.944273 32687 hybrid_clock.cc:648] HybridClock initialized: now 1786085321944218 us; error 50 us; skew 500 ppm
I20260807 06:48:41.944633 32687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:41 dist-test-slave-3kk6 krb5kdc[32331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:41.948452 32687 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:41.950127 32687 webserver.cc:460] Webserver started at http://127.27.54.66:45917/ using document root <none> and password file <none>
I20260807 06:48:41.950987 32687 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:41.951154 32687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:41.951495 32687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:41.954787 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "44d5e890324949dd8b6af3b4b8528bdb"
format_stamp: "Formatted at 2026-08-07 06:48:41 on dist-test-slave-3kk6"
server_key: "4187d94e3b6605652bf28ddbc3e251ac"
server_key_iv: "567814efc19b892d77409d630c135aed"
server_key_version: "encryptionkey@0"
I20260807 06:48:41.955703 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "44d5e890324949dd8b6af3b4b8528bdb"
format_stamp: "Formatted at 2026-08-07 06:48:41 on dist-test-slave-3kk6"
server_key: "4187d94e3b6605652bf28ddbc3e251ac"
server_key_iv: "567814efc19b892d77409d630c135aed"
server_key_version: "encryptionkey@0"
I20260807 06:48:41.961385 32687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 06:48:41.965296 32707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:41.966887 32687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:48:41.967082 32687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "44d5e890324949dd8b6af3b4b8528bdb"
format_stamp: "Formatted at 2026-08-07 06:48:41 on dist-test-slave-3kk6"
server_key: "4187d94e3b6605652bf28ddbc3e251ac"
server_key_iv: "567814efc19b892d77409d630c135aed"
server_key_version: "encryptionkey@0"
I20260807 06:48:41.967310 32687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:41.988050 32687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:41.993177 32687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:41.993505 32687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:41.994603 32687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:41.996182 32687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:41.996258 32687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:41.996348 32687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:41.996394 32687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:42.032615 32687 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:39985
I20260807 06:48:42.032644 408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:39985 every 8 connection(s)
I20260807 06:48:42.034021 32687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:42.040602 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 32687
I20260807 06:48:42.040865 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:42.041371 27865 external_mini_cluster.cc:1428] Setting key 6badf364114c2f4f01d8a7f1e9c87b86
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:42.055969 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.036754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:46293 (local address 127.27.54.126:44955)
0807 06:48:42.037227 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:42.037232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:42.038110 (+ 878us) server_negotiation.cc:407] Connection header received
0807 06:48:42.040007 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:42.040013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:42.040107 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:42.040251 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:42.042104 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.043095 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.044913 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.045364 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.045508 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.049361 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:42.049367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:42.049370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:42.049405 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:42.052399 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.053248 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.053253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.053256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.053453 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.053985 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.053990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.053996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.054313 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:42.054490 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:42.055325 (+ 835us) server_negotiation.cc:299] Negotiation successful
0807 06:48:42.055579 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":260,"threads_started":1}
I20260807 06:48:42.057106 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.037185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:44955 (local address 127.27.54.66:46293)
0807 06:48:42.037922 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:42.037967 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:42.039705 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:42.040514 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:42.040522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:42.041204 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:42.041867 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:42.041886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.043322 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.043328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:42.044727 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:42.044742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.045630 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.045638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:42.045885 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.046716 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:42.046758 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:42.049097 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:42.052599 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:42.052608 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:42.052623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:42.053103 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:42.053588 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:42.053592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:42.053594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:42.053848 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:42.054462 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:42.054471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:42.054930 (+ 459us) client_negotiation.cc:769] Sending connection context
0807 06:48:42.055235 (+ 305us) client_negotiation.cc:240] Negotiation successful
0807 06:48:42.055850 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":278,"threads_started":1}
I20260807 06:48:42.058904 411 heartbeater.cc:347] Connected to a master server at 127.27.54.126:44955
I20260807 06:48:42.059257 411 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:42.060082 411 heartbeater.cc:511] Master 127.27.54.126:44955 requested a full tablet report, sending...
I20260807 06:48:42.061861 32398 ts_manager.cc:194] Registered new tserver with Master: 44d5e890324949dd8b6af3b4b8528bdb (127.27.54.66:39985)
I20260807 06:48:42.063216 32398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:46293
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:42.103109 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:44955
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:42.463268 419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:42.463624 419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:42.463798 419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:42.472043 419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:42.472185 419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:42.472384 419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:42.482903 419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:42.484275 419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:42.485690 419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:42.500952 428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:42.501185 426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:42.500923 424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:42.502285 419 server_base.cc:1034] running on GCE node
I20260807 06:48:42.503118 419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:42.504256 419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:42.505491 419 hybrid_clock.cc:648] HybridClock initialized: now 1786085322505434 us; error 83 us; skew 500 ppm
I20260807 06:48:42.505832 419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:42.509513 419 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:42.511242 419 webserver.cc:460] Webserver started at http://127.27.54.67:35563/ using document root <none> and password file <none>
I20260807 06:48:42.511998 419 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:42.512161 419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:42.512501 419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:42.515882 419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "d0d3bfad459d42d1b1610e1305208d6c"
format_stamp: "Formatted at 2026-08-07 06:48:42 on dist-test-slave-3kk6"
server_key: "072fa30b3541366456670410d3a7d0a0"
server_key_iv: "9e697f06515234b2955a03f80a4dac19"
server_key_version: "encryptionkey@0"
I20260807 06:48:42.516783 419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "d0d3bfad459d42d1b1610e1305208d6c"
format_stamp: "Formatted at 2026-08-07 06:48:42 on dist-test-slave-3kk6"
server_key: "072fa30b3541366456670410d3a7d0a0"
server_key_iv: "9e697f06515234b2955a03f80a4dac19"
server_key_version: "encryptionkey@0"
I20260807 06:48:42.518206 32680 heartbeater.cc:503] Master 127.27.54.126:44955 was elected leader, sending a full tablet report...
I20260807 06:48:42.523283 419 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 06:48:42.527165 438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:42.528672 419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:48:42.528884 419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "d0d3bfad459d42d1b1610e1305208d6c"
format_stamp: "Formatted at 2026-08-07 06:48:42 on dist-test-slave-3kk6"
server_key: "072fa30b3541366456670410d3a7d0a0"
server_key_iv: "9e697f06515234b2955a03f80a4dac19"
server_key_version: "encryptionkey@0"
I20260807 06:48:42.529117 419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:42.544900 419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:42.550076 419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:42.550442 419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:42.551698 419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:42.553207 419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:42.553321 419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:42.553488 419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:42.553570 419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:42.590492 419 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:37525
I20260807 06:48:42.590591 611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:37525 every 8 connection(s)
I20260807 06:48:42.592001 419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:42.600972 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 419
I20260807 06:48:42.601200 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:42.601804 27865 external_mini_cluster.cc:1428] Setting key 2d0589211f6b1c4e7c4d2e3af98dfa8a
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:42.614024 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.594839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:54433 (local address 127.27.54.126:44955)
0807 06:48:42.595301 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:42.595307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:42.596355 (+ 1048us) server_negotiation.cc:407] Connection header received
0807 06:48:42.598210 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:42.598217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:42.598303 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:42.598420 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:42.600256 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.601092 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.603402 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.603981 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.604130 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.608123 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:42.608128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:42.608131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:42.608166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:42.610633 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.611486 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.611492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.611496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.611655 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.612128 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.612135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.612144 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.612437 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:42.612604 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:42.613357 (+ 753us) server_negotiation.cc:299] Negotiation successful
0807 06:48:42.613599 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":210,"threads_started":1}
I20260807 06:48:42.615108 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.595453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:44955 (local address 127.27.54.67:54433)
0807 06:48:42.596169 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:42.596209 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:42.597894 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:42.598644 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:42.598702 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:42.599353 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:42.600068 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:42.600088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.601343 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.601349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:42.603254 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:42.603267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.604246 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.604257 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:42.604604 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.605480 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:42.605505 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:42.607870 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:42.610840 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:42.610850 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:42.610874 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:42.611334 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:42.611776 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:42.611782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:42.611785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:42.612000 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:42.612580 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:42.612589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:42.612960 (+ 371us) client_negotiation.cc:769] Sending connection context
0807 06:48:42.613242 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 06:48:42.613848 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":286,"threads_started":1}
I20260807 06:48:42.616719 612 heartbeater.cc:347] Connected to a master server at 127.27.54.126:44955
I20260807 06:48:42.617095 612 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:42.618019 612 heartbeater.cc:511] Master 127.27.54.126:44955 requested a full tablet report, sending...
I20260807 06:48:42.619598 32398 ts_manager.cc:194] Registered new tserver with Master: d0d3bfad459d42d1b1610e1305208d6c (127.27.54.67:37525)
I20260807 06:48:42.620471 32398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:54433
I20260807 06:48:42.630869 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:42.695909 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.680183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.27.54.126:44955)
0807 06:48:42.680394 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:42.680399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:42.680916 (+ 517us) server_negotiation.cc:407] Connection header received
0807 06:48:42.681077 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:42.681081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:42.681167 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:42.681273 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:42.682844 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.683598 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.684940 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.685420 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.685544 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.689883 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:42.689889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:42.689893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:42.689935 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:42.692874 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.693592 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.693606 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.693608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.693763 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.694243 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.694248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.694254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.694710 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:42.695016 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:42.695304 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 06:48:42.695571 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260807 06:48:42.696573 32398 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48684
I20260807 06:48:42.698799 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 32482
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:42.718842 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.700120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.27.54.65:45151)
0807 06:48:42.701009 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:42.701017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:42.701046 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:48:42.701120 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:42.701127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:42.701351 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:42.701590 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:42.703054 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.704238 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.705801 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.706452 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.706572 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.711534 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:42.711553 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:42.711570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:42.711620 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:42.715283 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.716190 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.716199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.716202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.716363 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.716841 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.716851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.716854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.717259 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:42.717623 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:42.717972 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 06:48:42.718279 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":484,"threads_started":1}
W20260807 06:48:42.720278 32673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35594
W20260807 06:48:42.721580 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 32482: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:42.721819 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 32482
I20260807 06:48:42.755636 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 32687
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:42.776857 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.756701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.27.54.66:39985)
0807 06:48:42.757270 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:42.757278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:42.757306 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:48:42.757374 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:42.757381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:42.757604 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:42.757739 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:42.759544 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.760585 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.762521 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.763267 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.763385 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.769425 (+ 6040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:42.769445 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:42.769463 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:42.769511 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:42.773710 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.774634 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.774720 (+ 86us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.774723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.774880 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.775253 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.775261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.775263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.775703 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:42.775965 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:42.776282 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 06:48:42.776509 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":299,"threads_started":1}
W20260807 06:48:42.778752 404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50184
W20260807 06:48:42.780004 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 32687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:42.780208 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 32687
I20260807 06:48:42.811375 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 419
Aug 07 06:48:42 dist-test-slave-3kk6 krb5kdc[32331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:42.831763 629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:42.812517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.27.54.67:37525)
0807 06:48:42.813244 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:42.813251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:42.813283 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:48:42.813348 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:42.813356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:42.813575 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:42.813711 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:42.815331 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.816453 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.817721 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:42.818293 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:42.818402 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:42.824112 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:42.824129 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:42.824145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:42.824185 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:42.828122 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.829298 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.829308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.829310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.829443 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:42.829846 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:42.829855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:42.829857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:42.830261 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:42.830516 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:42.831023 (+ 507us) server_negotiation.cc:299] Negotiation successful
0807 06:48:42.831267 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":394,"threads_started":1}
W20260807 06:48:42.833375 603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59984
W20260807 06:48:42.834985 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 419: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:48:42.835151 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 419
I20260807 06:48:42.861804 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 32349
I20260807 06:48:42.863288 32439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:42.987895 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 32349
2026-08-07T06:48:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4287 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:43 dist-test-slave-3kk6 krb5kdc[635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:43 dist-test-slave-3kk6 krb5kdc[635](info): set up 2 sockets
Aug 07 06:48:43 dist-test-slave-3kk6 krb5kdc[635](info): commencing operation
krb5kdc: starting...
W20260807 06:48:44.771262 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.732s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:45 dist-test-slave-3kk6 krb5kdc[635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:45.197291 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39849
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:45.746048 653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:45.746450 653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:45.746630 653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:45.754107 653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:45.754298 653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:45.754400 653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:45.754514 653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:45.754609 653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:45.764971 653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:39849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:45.766500 653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:45.767930 653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:45.780839 663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:45.780908 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:45.780884 661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:45.781772 653 server_base.cc:1034] running on GCE node
I20260807 06:48:45.782732 653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:45.784307 653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:45.785563 653 hybrid_clock.cc:648] HybridClock initialized: now 1786085325785523 us; error 56 us; skew 500 ppm
I20260807 06:48:45.785950 653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:45 dist-test-slave-3kk6 krb5kdc[635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085325, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:45.794461 653 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:45.796327 653 webserver.cc:460] Webserver started at http://127.27.54.126:36075/ using document root <none> and password file <none>
I20260807 06:48:45.797115 653 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:45.797281 653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:45.797633 653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:45.801028 653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "ca0bd6663ae54feb9369e74914ad71b3"
format_stamp: "Formatted at 2026-08-07 06:48:45 on dist-test-slave-3kk6"
server_key: "0cac7ae2aaf865ac4fc6040b7ac88d6f"
server_key_iv: "7af5a5e9f5b63b7f0d31a008f4473051"
server_key_version: "encryptionkey@0"
I20260807 06:48:45.801927 653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "ca0bd6663ae54feb9369e74914ad71b3"
format_stamp: "Formatted at 2026-08-07 06:48:45 on dist-test-slave-3kk6"
server_key: "0cac7ae2aaf865ac4fc6040b7ac88d6f"
server_key_iv: "7af5a5e9f5b63b7f0d31a008f4473051"
server_key_version: "encryptionkey@0"
I20260807 06:48:45.807546 653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 06:48:45.811439 676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:45.812951 653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 06:48:45.813166 653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "ca0bd6663ae54feb9369e74914ad71b3"
format_stamp: "Formatted at 2026-08-07 06:48:45 on dist-test-slave-3kk6"
server_key: "0cac7ae2aaf865ac4fc6040b7ac88d6f"
server_key_iv: "7af5a5e9f5b63b7f0d31a008f4473051"
server_key_version: "encryptionkey@0"
I20260807 06:48:45.813398 653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:45.831529 653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:45.836944 653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:45.837278 653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:45.858088 653 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:39849
I20260807 06:48:45.858124 755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:39849 every 8 connection(s)
I20260807 06:48:45.859720 653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:45.860303 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 653
I20260807 06:48:45.860528 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:45.861388 27865 external_mini_cluster.cc:1428] Setting key 268650c880d24f8665ec2e2150e2a745
I20260807 06:48:45.864322 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:48:45 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:45.878296 756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3: Bootstrap starting.
I20260807 06:48:45.881999 756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:45.883179 756 log.cc:824] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:45.885926 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:45.863604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.27.54.126:39849)
0807 06:48:45.864489 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:45.864500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:45.864542 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:48:45.866266 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:45.866287 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:45.866753 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:45.867116 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:45.869139 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:45.870578 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:45.872458 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:45.873139 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:45.873272 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:45.878276 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:45.878288 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:45.878304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:45.878356 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:45.881612 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:45.882447 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:45.882459 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:45.882466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:45.882641 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:45.883122 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:45.883128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:45.883132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:45.883742 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:45.884069 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:45.884560 (+ 491us) server_negotiation.cc:299] Negotiation successful
0807 06:48:45.884968 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":410,"threads_started":1}
I20260807 06:48:45.886324 756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3: No bootstrap required, opened a new log
I20260807 06:48:45.896102 756 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0bd6663ae54feb9369e74914ad71b3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39849 } }
I20260807 06:48:45.896569 756 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:45.896693 756 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0bd6663ae54feb9369e74914ad71b3, State: Initialized, Role: FOLLOWER
I20260807 06:48:45.897356 756 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0bd6663ae54feb9369e74914ad71b3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39849 } }
I20260807 06:48:45.897612 756 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:45.897773 756 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:45.897974 756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:45.900375 756 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0bd6663ae54feb9369e74914ad71b3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39849 } }
I20260807 06:48:45.900971 756 leader_election.cc:302] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0bd6663ae54feb9369e74914ad71b3; no voters:
I20260807 06:48:45.901734 756 leader_election.cc:288] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:45.901858 766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:45.902523 766 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [term 1 LEADER]: Becoming Leader. State: Replica: ca0bd6663ae54feb9369e74914ad71b3, State: Running, Role: LEADER
I20260807 06:48:45.903193 766 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0bd6663ae54feb9369e74914ad71b3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39849 } }
I20260807 06:48:45.904203 756 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:45.906384 767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0bd6663ae54feb9369e74914ad71b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0bd6663ae54feb9369e74914ad71b3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39849 } } }
I20260807 06:48:45.906453 769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0bd6663ae54feb9369e74914ad71b3. Latest consensus state: current_term: 1 leader_uuid: "ca0bd6663ae54feb9369e74914ad71b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0bd6663ae54feb9369e74914ad71b3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39849 } } }
I20260807 06:48:45.906803 767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:45.906890 769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:45.908666 773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:45.915083 773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:45.924602 773 catalog_manager.cc:1347] Generated new cluster ID: 10cf6dc950f04b589d690cd5430538be
I20260807 06:48:45.924782 773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:45.947072 773 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:45.948566 773 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:45.957620 773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0bd6663ae54feb9369e74914ad71b3: Generated new TSK 0
I20260807 06:48:45.958536 773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:46.024773 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39849
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:46.384292 789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:46.384661 789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:46.384871 789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:46.392695 789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:46.392897 789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:46.393090 789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:46.403762 789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:46.405331 789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:46.406786 789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:46.421273 795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:46.421339 800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:46.421347 796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:46.422523 789 server_base.cc:1034] running on GCE node
I20260807 06:48:46.423270 789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:46.424304 789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:46.425551 789 hybrid_clock.cc:648] HybridClock initialized: now 1786085326425508 us; error 57 us; skew 500 ppm
I20260807 06:48:46.425884 789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:46 dist-test-slave-3kk6 krb5kdc[635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:46.432375 789 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:46.433871 789 webserver.cc:460] Webserver started at http://127.27.54.65:46879/ using document root <none> and password file <none>
I20260807 06:48:46.434548 789 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:46.434710 789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:46.435017 789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:46.438166 789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "ed8a995ba8044002a01df02408398aca"
format_stamp: "Formatted at 2026-08-07 06:48:46 on dist-test-slave-3kk6"
server_key: "5a48f0b1971c7055039ad5a37185bb40"
server_key_iv: "3032c0a0c44761858a2092ce47d6b816"
server_key_version: "encryptionkey@0"
I20260807 06:48:46.438998 789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "ed8a995ba8044002a01df02408398aca"
format_stamp: "Formatted at 2026-08-07 06:48:46 on dist-test-slave-3kk6"
server_key: "5a48f0b1971c7055039ad5a37185bb40"
server_key_iv: "3032c0a0c44761858a2092ce47d6b816"
server_key_version: "encryptionkey@0"
I20260807 06:48:46.444183 789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:48:46.447631 810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:46.448905 789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:48:46.449075 789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "ed8a995ba8044002a01df02408398aca"
format_stamp: "Formatted at 2026-08-07 06:48:46 on dist-test-slave-3kk6"
server_key: "5a48f0b1971c7055039ad5a37185bb40"
server_key_iv: "3032c0a0c44761858a2092ce47d6b816"
server_key_version: "encryptionkey@0"
I20260807 06:48:46.449277 789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:46.468091 789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:46.472333 789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:46.472661 789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:46.473693 789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:46.475435 789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:46.475519 789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:46.475725 789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:46.475787 789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:46.511684 789 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:45523
I20260807 06:48:46.511725 971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:45523 every 8 connection(s)
I20260807 06:48:46.513044 789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:46.522938 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 789
I20260807 06:48:46.523191 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:46.524006 27865 external_mini_cluster.cc:1428] Setting key 7062da9bbd365a7f29b0ff895baf916a
Aug 07 06:48:46 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:46.537387 976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:46.515912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:37953 (local address 127.27.54.126:39849)
0807 06:48:46.516373 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:46.516377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:46.517460 (+ 1083us) server_negotiation.cc:407] Connection header received
0807 06:48:46.519473 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:46.519477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:46.519548 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:46.519657 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:46.521143 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:46.521859 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:46.523932 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:46.524626 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:46.524824 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:46.528706 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:46.528714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:46.528722 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:46.528769 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:46.532794 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:46.533960 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:46.533965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:46.533967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:46.534182 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:46.534867 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:46.534872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:46.534876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:46.535260 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:46.535432 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:46.536616 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0807 06:48:46.536920 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260807 06:48:46.539142 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:46.516519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39849 (local address 127.27.54.65:37953)
0807 06:48:46.517299 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:46.517342 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:46.519210 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:46.519876 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:46.519883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:46.520432 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:46.520987 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:46.521005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:46.522100 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:46.522104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:46.523757 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:46.523771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:46.524914 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:46.524924 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:46.525222 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:46.526023 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:46.526058 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:46.528467 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:46.533007 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:46.533020 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:46.533040 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:46.533762 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:46.534397 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:46.534404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:46.534407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:46.534713 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:46.535519 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:46.535528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:46.536063 (+ 535us) client_negotiation.cc:769] Sending connection context
0807 06:48:46.536452 (+ 389us) client_negotiation.cc:240] Negotiation successful
0807 06:48:46.537117 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":271,"threads_started":1}
I20260807 06:48:46.541286 973 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39849
I20260807 06:48:46.541702 973 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:46.542814 973 heartbeater.cc:511] Master 127.27.54.126:39849 requested a full tablet report, sending...
I20260807 06:48:46.545979 701 ts_manager.cc:194] Registered new tserver with Master: ed8a995ba8044002a01df02408398aca (127.27.54.65:45523)
I20260807 06:48:46.547843 701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:37953
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:46.594708 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39849
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:46.959668 982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:46.960005 982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:46.960104 982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:46.967793 982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:46.967945 982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:46.968066 982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:46.978437 982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:46.979900 982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:46.981328 982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:46.995536 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:46.995584 989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:46.995855 987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:46.997118 982 server_base.cc:1034] running on GCE node
I20260807 06:48:46.997879 982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:46.999029 982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:47.000293 982 hybrid_clock.cc:648] HybridClock initialized: now 1786085327000252 us; error 49 us; skew 500 ppm
I20260807 06:48:47.000653 982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:47.004257 982 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:47.006021 982 webserver.cc:460] Webserver started at http://127.27.54.66:33217/ using document root <none> and password file <none>
I20260807 06:48:47.006829 982 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:47.006991 982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:47.007337 982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:47.010752 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "3bb26275bf0a4ba08b79cc9d9b5ef3bc"
format_stamp: "Formatted at 2026-08-07 06:48:47 on dist-test-slave-3kk6"
server_key: "e888544339c5f72b758a228ed428640f"
server_key_iv: "46140771ff1db805ddcbdee8c5cddd48"
server_key_version: "encryptionkey@0"
I20260807 06:48:47.011706 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "3bb26275bf0a4ba08b79cc9d9b5ef3bc"
format_stamp: "Formatted at 2026-08-07 06:48:47 on dist-test-slave-3kk6"
server_key: "e888544339c5f72b758a228ed428640f"
server_key_iv: "46140771ff1db805ddcbdee8c5cddd48"
server_key_version: "encryptionkey@0"
I20260807 06:48:47.017591 982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:48:47.021543 1001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:47.023183 982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:47.023391 982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "3bb26275bf0a4ba08b79cc9d9b5ef3bc"
format_stamp: "Formatted at 2026-08-07 06:48:47 on dist-test-slave-3kk6"
server_key: "e888544339c5f72b758a228ed428640f"
server_key_iv: "46140771ff1db805ddcbdee8c5cddd48"
server_key_version: "encryptionkey@0"
I20260807 06:48:47.023638 982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:47.053164 982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:47.058365 982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:47.058753 982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:47.059933 982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:47.061480 982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:47.061594 982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:47.061766 982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:47.061844 982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:47.099335 982 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:42893
I20260807 06:48:47.099400 1174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:42893 every 8 connection(s)
I20260807 06:48:47.100781 982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:47.104249 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 982
I20260807 06:48:47.104446 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:47.104998 27865 external_mini_cluster.cc:1428] Setting key c2a27e6913efdd015fa008a4fe024e25
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:47.124339 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.103704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:58611 (local address 127.27.54.126:39849)
0807 06:48:47.104272 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:47.104280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:47.105541 (+ 1261us) server_negotiation.cc:407] Connection header received
0807 06:48:47.107866 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:47.107872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:47.107962 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:47.108095 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:47.109854 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.110592 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.112245 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.112700 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.112845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.116793 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:47.116798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:47.116802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:47.116837 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:47.120732 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.121538 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.121546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.121552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.121720 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.122175 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.122183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.122190 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.122587 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:47.122777 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:47.123544 (+ 767us) server_negotiation.cc:299] Negotiation successful
0807 06:48:47.123817 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260807 06:48:47.125058 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.104321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39849 (local address 127.27.54.66:58611)
0807 06:48:47.105334 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:47.105392 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:47.107585 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:47.108350 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:47.108360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:47.109009 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:47.109655 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:47.109675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.110800 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.110808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:47.112084 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:47.112097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.112966 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.112974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:47.113219 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.113998 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:47.114029 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:47.116487 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:47.120917 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:47.120925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:47.120939 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:47.121398 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:47.121837 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:47.121840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:47.121842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:47.122054 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:47.122760 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:47.122766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:47.123131 (+ 365us) client_negotiation.cc:769] Sending connection context
0807 06:48:47.123388 (+ 257us) client_negotiation.cc:240] Negotiation successful
0807 06:48:47.123807 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":492,"threads_started":1}
I20260807 06:48:47.126735 1175 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39849
I20260807 06:48:47.127089 1175 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:47.127985 1175 heartbeater.cc:511] Master 127.27.54.126:39849 requested a full tablet report, sending...
I20260807 06:48:47.129698 701 ts_manager.cc:194] Registered new tserver with Master: 3bb26275bf0a4ba08b79cc9d9b5ef3bc (127.27.54.66:42893)
I20260807 06:48:47.130587 701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:58611
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:47.166746 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39849
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:47.531893 1181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:47.532291 1181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:47.532483 1181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:47.540282 1181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:47.540474 1181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:47.540674 1181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:47.551108 1181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:47.552522 973 heartbeater.cc:503] Master 127.27.54.126:39849 was elected leader, sending a full tablet report...
I20260807 06:48:47.552983 1181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:47.554738 1181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:47.569417 1193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:47.569427 1194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:47.569240 1196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:47.570765 1181 server_base.cc:1034] running on GCE node
I20260807 06:48:47.571470 1181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:47.572562 1181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:47.573836 1181 hybrid_clock.cc:648] HybridClock initialized: now 1786085327573754 us; error 77 us; skew 500 ppm
I20260807 06:48:47.574211 1181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:47.577876 1181 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:47.579757 1181 webserver.cc:460] Webserver started at http://127.27.54.67:34287/ using document root <none> and password file <none>
I20260807 06:48:47.580543 1181 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:47.580713 1181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:47.581079 1181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:47.584509 1181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "02ea3c7fbbff45fc950ffa07e10dffa3"
format_stamp: "Formatted at 2026-08-07 06:48:47 on dist-test-slave-3kk6"
server_key: "40ef99bfc31cd47ca495a1c99b1d65be"
server_key_iv: "20ea5d26b4b0dc51f118361aa0e0dc9e"
server_key_version: "encryptionkey@0"
I20260807 06:48:47.585455 1181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "02ea3c7fbbff45fc950ffa07e10dffa3"
format_stamp: "Formatted at 2026-08-07 06:48:47 on dist-test-slave-3kk6"
server_key: "40ef99bfc31cd47ca495a1c99b1d65be"
server_key_iv: "20ea5d26b4b0dc51f118361aa0e0dc9e"
server_key_version: "encryptionkey@0"
I20260807 06:48:47.591511 1181 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 06:48:47.595702 1206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:47.597325 1181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260807 06:48:47.597553 1181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "02ea3c7fbbff45fc950ffa07e10dffa3"
format_stamp: "Formatted at 2026-08-07 06:48:47 on dist-test-slave-3kk6"
server_key: "40ef99bfc31cd47ca495a1c99b1d65be"
server_key_iv: "20ea5d26b4b0dc51f118361aa0e0dc9e"
server_key_version: "encryptionkey@0"
I20260807 06:48:47.597810 1181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:47.633073 1181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:47.637822 1181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:47.638159 1181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:47.639292 1181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:47.640692 1181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:47.640772 1181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:47.640869 1181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:47.640916 1181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:47.677369 1181 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:40155
I20260807 06:48:47.677423 1379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:40155 every 8 connection(s)
I20260807 06:48:47.678984 1181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:47.685326 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 1181
I20260807 06:48:47.685560 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:47.686141 27865 external_mini_cluster.cc:1428] Setting key 6ac5b395e936fe568ebf8be3b1374f94
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:47.702175 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.681842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:50953 (local address 127.27.54.126:39849)
0807 06:48:47.682369 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:47.682373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:47.683475 (+ 1102us) server_negotiation.cc:407] Connection header received
0807 06:48:47.685495 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:47.685503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:47.685613 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:47.685764 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:47.687725 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.688845 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.691046 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.691467 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.691568 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.695531 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:47.695537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:47.695540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:47.695578 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:47.698328 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.699307 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.699312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.699315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.699459 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.699921 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.699924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.699927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.700302 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:47.700480 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:47.701537 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0807 06:48:47.701790 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":256,"threads_started":1}
I20260807 06:48:47.703470 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.682471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39849 (local address 127.27.54.67:50953)
0807 06:48:47.683279 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:47.683322 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:47.685151 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:47.686016 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:47.686025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:47.686801 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:47.687517 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:47.687538 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.689064 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.689070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:47.690883 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:47.690898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.691733 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.691739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:47.691953 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.692681 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:47.692711 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:47.695262 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:47.698515 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:47.698523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:47.698538 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:47.699146 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:47.699564 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:47.699567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:47.699569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:47.699805 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:47.700462 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:47.700468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:47.700988 (+ 520us) client_negotiation.cc:769] Sending connection context
0807 06:48:47.701353 (+ 365us) client_negotiation.cc:240] Negotiation successful
0807 06:48:47.701959 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":380,"threads_started":1}
I20260807 06:48:47.704962 1383 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39849
I20260807 06:48:47.705350 1383 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:47.706133 1383 heartbeater.cc:511] Master 127.27.54.126:39849 requested a full tablet report, sending...
I20260807 06:48:47.707755 701 ts_manager.cc:194] Registered new tserver with Master: 02ea3c7fbbff45fc950ffa07e10dffa3 (127.27.54.67:40155)
I20260807 06:48:47.708581 701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:50953
I20260807 06:48:47.715454 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:47.778879 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.762552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.27.54.65:45523)
0807 06:48:47.763488 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:47.763495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:47.763523 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:48:47.763590 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:47.763597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:47.763804 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:47.763935 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:47.765409 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.766470 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.767737 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.768323 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.768445 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.772685 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:47.772699 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:47.772714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:47.772751 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:47.775943 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.776686 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.776694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.776697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.776825 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.777193 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.777196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.777198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.777565 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:47.777809 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:47.778135 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 06:48:47.778343 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":587,"threads_started":1}
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:47.797668 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.783736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.27.54.126:39849)
0807 06:48:47.783976 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:47.783981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:47.784068 (+ 87us) server_negotiation.cc:407] Connection header received
0807 06:48:47.784205 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:47.784209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:47.784277 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:47.784372 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:47.785656 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.786412 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.787642 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.788025 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.788120 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.792252 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:47.792257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:47.792261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:47.792297 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:47.795115 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.795771 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.795775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.795777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.795916 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.796264 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.796267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.796269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.796618 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:47.796810 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:47.797141 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 06:48:47.797319 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 06:48:47.798296 701 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778
I20260807 06:48:47.799513 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 789
I20260807 06:48:47.813074 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.800617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.27.54.65:45523)
0807 06:48:47.800838 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:47.800843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:47.800878 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:48:47.801045 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:47.801049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:47.801116 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:47.801196 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:47.802532 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.803407 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.804790 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.805235 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.805358 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.807853 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:47.807858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:47.807862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:47.807896 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:47.810598 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.811330 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.811334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.811336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.811465 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.811799 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.811802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.811804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.812066 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:47.812249 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:47.812521 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 06:48:47.812703 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:48:47.813788 966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:47.901551 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 789
I20260807 06:48:47.927208 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 982
Aug 07 06:48:47 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:47.944916 1399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:47.928382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.27.54.66:42893)
0807 06:48:47.928979 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:47.928986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:47.929012 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:48:47.929083 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:47.929089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:47.929309 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:47.929495 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:47.930912 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.931962 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.933279 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:47.933850 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:47.934003 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:47.938635 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:47.938873 (+ 238us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:47.938892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:47.938936 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:47.941928 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.942910 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.942917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.942920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.943054 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:47.943444 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:47.943447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:47.943449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:47.943801 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:47.944062 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:47.944380 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 06:48:47.944592 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":326,"threads_started":1}
I20260807 06:48:47.946398 1166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:48.032406 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 982
I20260807 06:48:48.059154 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 1181
Aug 07 06:48:48 dist-test-slave-3kk6 krb5kdc[635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:48.077207 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:48.060325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.27.54.67:40155)
0807 06:48:48.061046 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:48.061053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:48.061083 (+ 30us) server_negotiation.cc:407] Connection header received
0807 06:48:48.061150 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:48.061156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:48.061380 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:48.061522 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:48.062893 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:48.063942 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:48.065370 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:48.065963 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:48.066123 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:48.070901 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:48.070916 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:48.070931 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:48.070971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:48.073961 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:48.074831 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:48.074839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:48.074842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:48.074989 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:48.075441 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:48.075445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:48.075447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:48.075827 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:48.076209 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:48.076531 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 06:48:48.076824 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
I20260807 06:48:48.078724 1375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:48.169287 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 1181
I20260807 06:48:48.196250 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 653
I20260807 06:48:48.197597 747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:48.306362 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 653
2026-08-07T06:48:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5319 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:48:48 dist-test-slave-3kk6 krb5kdc[1409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:48:48 dist-test-slave-3kk6 krb5kdc[1409](info): set up 2 sockets
Aug 07 06:48:48 dist-test-slave-3kk6 krb5kdc[1409](info): commencing operation
krb5kdc: starting...
W20260807 06:48:49.633277 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:48:49 dist-test-slave-3kk6 krb5kdc[1409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:48:49.900436 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:42721
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:50.261302 1428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:50.261704 1428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:50.261878 1428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:50.269178 1428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:50.269331 1428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:50.269443 1428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:50.269552 1428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:50.269646 1428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:50.279515 1428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:50.280891 1428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:50.282254 1428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:50.294029 1435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:50.294277 1433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:50.294071 1439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:50.295217 1428 server_base.cc:1034] running on GCE node
I20260807 06:48:50.296068 1428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:50.297689 1428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:50.298938 1428 hybrid_clock.cc:648] HybridClock initialized: now 1786085330298889 us; error 53 us; skew 500 ppm
I20260807 06:48:50.299316 1428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:50 dist-test-slave-3kk6 krb5kdc[1409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:50.302618 1428 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:48:50.303906 1428 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:50.304049 1428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:50.304407 1428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:50.307652 1428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "3c0fa9a49d6d4250a30d9b3911968b66"
format_stamp: "Formatted at 2026-08-07 06:48:50 on dist-test-slave-3kk6"
server_key: "6b05a2514a7d10c4dfb9b76c835e3522"
server_key_iv: "c4c4677173e817313ad9e3f96249a5d7"
server_key_version: "encryptionkey@0"
I20260807 06:48:50.308569 1428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "3c0fa9a49d6d4250a30d9b3911968b66"
format_stamp: "Formatted at 2026-08-07 06:48:50 on dist-test-slave-3kk6"
server_key: "6b05a2514a7d10c4dfb9b76c835e3522"
server_key_iv: "c4c4677173e817313ad9e3f96249a5d7"
server_key_version: "encryptionkey@0"
I20260807 06:48:50.314110 1428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 06:48:50.318146 1446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:50.319607 1428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 06:48:50.319813 1428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "3c0fa9a49d6d4250a30d9b3911968b66"
format_stamp: "Formatted at 2026-08-07 06:48:50 on dist-test-slave-3kk6"
server_key: "6b05a2514a7d10c4dfb9b76c835e3522"
server_key_iv: "c4c4677173e817313ad9e3f96249a5d7"
server_key_version: "encryptionkey@0"
I20260807 06:48:50.320082 1428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:50.341950 1428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:50.346355 1428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:50.346673 1428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:50.366125 1428 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:42721
I20260807 06:48:50.366147 1519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:42721 every 8 connection(s)
I20260807 06:48:50.367231 1428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:50.368415 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 1428
I20260807 06:48:50.368644 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:50.369489 27865 external_mini_cluster.cc:1428] Setting key 412f887b60573aeef5939d46a9741f08
I20260807 06:48:50.372406 1520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:48:50 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:50.386363 1520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66: Bootstrap starting.
I20260807 06:48:50.390177 1520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:50.391595 1520 log.cc:824] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:50.395041 1520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66: No bootstrap required, opened a new log
I20260807 06:48:50.395150 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:50.371832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.27.54.126:42721)
0807 06:48:50.372703 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:50.372726 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:50.372779 (+ 53us) server_negotiation.cc:407] Connection header received
0807 06:48:50.375301 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:50.375330 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:50.375872 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:50.376362 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:50.377836 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:50.379929 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:50.381223 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:50.382278 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:50.382492 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:50.387135 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:50.387143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:50.387160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:50.387206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:50.390469 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:50.391392 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:50.391401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:50.391408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:50.391578 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:50.392040 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:50.392043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:50.392045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:50.392889 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:50.393199 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:50.393631 (+ 432us) server_negotiation.cc:299] Negotiation successful
0807 06:48:50.394004 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":373,"threads_started":1}
I20260807 06:48:50.404000 1520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42721 } }
I20260807 06:48:50.404448 1520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:50.404585 1520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0fa9a49d6d4250a30d9b3911968b66, State: Initialized, Role: FOLLOWER
I20260807 06:48:50.405202 1520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42721 } }
I20260807 06:48:50.405465 1520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:50.405609 1520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:50.405792 1520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:50.408420 1520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42721 } }
I20260807 06:48:50.408960 1520 leader_election.cc:302] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c0fa9a49d6d4250a30d9b3911968b66; no voters:
I20260807 06:48:50.409586 1520 leader_election.cc:288] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:50.409736 1526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:50.410540 1526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0fa9a49d6d4250a30d9b3911968b66, State: Running, Role: LEADER
I20260807 06:48:50.411181 1526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42721 } }
I20260807 06:48:50.412014 1520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:50.414076 1528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c0fa9a49d6d4250a30d9b3911968b66. Latest consensus state: current_term: 1 leader_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42721 } } }
I20260807 06:48:50.414522 1528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:50.414981 1527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c0fa9a49d6d4250a30d9b3911968b66" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42721 } } }
I20260807 06:48:50.415370 1527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:50.417591 1536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:50.424738 1536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:50.433710 1536 catalog_manager.cc:1347] Generated new cluster ID: d3d436edaa6c44a8b529a46fbbc362c1
I20260807 06:48:50.433935 1536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:50.453881 1536 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:50.455451 1536 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:50.462208 1536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66: Generated new TSK 0
I20260807 06:48:50.463406 1536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:48:50.527823 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42721
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:50.880411 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:50.880805 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:50.881016 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:50.889120 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:50.889328 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:50.889529 1553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:50.900833 1553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:50.902412 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:50.903962 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:50.919307 1561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:50.919262 1562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:50.919721 1565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:50.920274 1553 server_base.cc:1034] running on GCE node
I20260807 06:48:50.920940 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:50.921986 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:50.923197 1553 hybrid_clock.cc:648] HybridClock initialized: now 1786085330923154 us; error 42 us; skew 500 ppm
I20260807 06:48:50.923565 1553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:50 dist-test-slave-3kk6 krb5kdc[1409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:50.929602 1553 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:48:50.930918 1553 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:50.931062 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:50.931409 1553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:50.934697 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "376a43d564b24c7d9f28e7da2e43fe9f"
format_stamp: "Formatted at 2026-08-07 06:48:50 on dist-test-slave-3kk6"
server_key: "dcb8b69554f9170c1fc7e429aa7637ca"
server_key_iv: "8c5d65c9b4033d18b32202c05dc21abe"
server_key_version: "encryptionkey@0"
I20260807 06:48:50.935595 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "376a43d564b24c7d9f28e7da2e43fe9f"
format_stamp: "Formatted at 2026-08-07 06:48:50 on dist-test-slave-3kk6"
server_key: "dcb8b69554f9170c1fc7e429aa7637ca"
server_key_iv: "8c5d65c9b4033d18b32202c05dc21abe"
server_key_version: "encryptionkey@0"
I20260807 06:48:50.941102 1553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 06:48:50.944813 1576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:50.946372 1553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:48:50.946602 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "376a43d564b24c7d9f28e7da2e43fe9f"
format_stamp: "Formatted at 2026-08-07 06:48:50 on dist-test-slave-3kk6"
server_key: "dcb8b69554f9170c1fc7e429aa7637ca"
server_key_iv: "8c5d65c9b4033d18b32202c05dc21abe"
server_key_version: "encryptionkey@0"
I20260807 06:48:50.946931 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:50.981953 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:50.986371 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:50.986734 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:50.987813 1553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:50.989234 1553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:50.989379 1553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:50.989519 1553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:50.989643 1553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:51.025668 1553 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:44575
I20260807 06:48:51.025725 1739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:44575 every 8 connection(s)
I20260807 06:48:51.026630 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:51.036911 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 1553
I20260807 06:48:51.037129 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:51.037981 27865 external_mini_cluster.cc:1428] Setting key f6929cbf7ed33d2635edce03805c1de0
Aug 07 06:48:51 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:51.048893 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:51.029354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:55437 (local address 127.27.54.126:42721)
0807 06:48:51.029843 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:51.029848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:51.030992 (+ 1144us) server_negotiation.cc:407] Connection header received
0807 06:48:51.032995 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:51.033002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:51.033102 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:51.033234 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:51.034881 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.035628 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.037810 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.038302 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.038433 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:51.042533 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:51.042539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:51.042542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:51.042576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:51.045304 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:51.046138 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:51.046142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:51.046145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:51.046310 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:51.046909 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:51.046913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:51.046915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:51.047194 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:51.047327 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:51.048430 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0807 06:48:51.048621 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":233,"threads_started":1}
I20260807 06:48:51.050347 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:51.029970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42721 (local address 127.27.54.65:55437)
0807 06:48:51.030810 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:51.030852 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:51.032686 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:51.033479 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:51.033488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:51.034048 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:51.034631 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:51.034711 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.035806 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.035818 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:51.037626 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:51.037646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.038574 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.038581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:51.038876 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:51.039628 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:51.039651 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:51.042283 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:51.045477 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:51.045486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:51.045505 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:51.045988 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:51.046472 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:51.046480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:51.046486 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:51.046779 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:51.047330 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:51.047341 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:51.047845 (+ 504us) client_negotiation.cc:769] Sending connection context
0807 06:48:51.048210 (+ 365us) client_negotiation.cc:240] Negotiation successful
0807 06:48:51.048840 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260807 06:48:51.052228 1740 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42721
I20260807 06:48:51.052590 1740 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:51.053422 1740 heartbeater.cc:511] Master 127.27.54.126:42721 requested a full tablet report, sending...
I20260807 06:48:51.055783 1476 ts_manager.cc:194] Registered new tserver with Master: 376a43d564b24c7d9f28e7da2e43fe9f (127.27.54.65:44575)
I20260807 06:48:51.057258 1476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:55437
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:48:51.099035 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42721
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:51.453948 1747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:51.454306 1747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:51.454493 1747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:51.461964 1747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:51.462157 1747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:51.462360 1747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:51.472863 1747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:51.474296 1747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:51.475808 1747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:51.491274 1752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:51.491209 1756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:51.491391 1753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:51.492049 1747 server_base.cc:1034] running on GCE node
I20260807 06:48:51.492733 1747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:51.493775 1747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:51.495011 1747 hybrid_clock.cc:648] HybridClock initialized: now 1786085331494958 us; error 51 us; skew 500 ppm
I20260807 06:48:51.495373 1747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:51 dist-test-slave-3kk6 krb5kdc[1409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:51.498772 1747 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:48:51.500049 1747 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:51.500149 1747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:51.500454 1747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:51.503605 1747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "ef30e8eb68f04a13867f55d939bab758"
format_stamp: "Formatted at 2026-08-07 06:48:51 on dist-test-slave-3kk6"
server_key: "ddee8b54b3f05177f5aee4a94f60c8ec"
server_key_iv: "d6fdac96cd29ce2eeb4165d52cab1953"
server_key_version: "encryptionkey@0"
I20260807 06:48:51.504437 1747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "ef30e8eb68f04a13867f55d939bab758"
format_stamp: "Formatted at 2026-08-07 06:48:51 on dist-test-slave-3kk6"
server_key: "ddee8b54b3f05177f5aee4a94f60c8ec"
server_key_iv: "d6fdac96cd29ce2eeb4165d52cab1953"
server_key_version: "encryptionkey@0"
I20260807 06:48:51.509809 1747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 06:48:51.513839 1770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:51.515331 1747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:51.515538 1747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "ef30e8eb68f04a13867f55d939bab758"
format_stamp: "Formatted at 2026-08-07 06:48:51 on dist-test-slave-3kk6"
server_key: "ddee8b54b3f05177f5aee4a94f60c8ec"
server_key_iv: "d6fdac96cd29ce2eeb4165d52cab1953"
server_key_version: "encryptionkey@0"
I20260807 06:48:51.515776 1747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:51.551555 1747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:51.556062 1747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:51.556382 1747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:51.557531 1747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:51.559005 1747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:51.559096 1747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:51.559183 1747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:51.559239 1747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:51.595451 1747 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:40621
I20260807 06:48:51.595552 1930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:40621 every 8 connection(s)
I20260807 06:48:51.596443 1747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:51.598063 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 1747
I20260807 06:48:51.598260 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:51.598868 27865 external_mini_cluster.cc:1428] Setting key f7c4a17e99da7b5ddf84ce83654ae2c6
Aug 07 06:48:51 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:51.619897 1934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:51.599468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:40339 (local address 127.27.54.126:42721)
0807 06:48:51.599961 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:51.599966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:51.601203 (+ 1237us) server_negotiation.cc:407] Connection header received
0807 06:48:51.603154 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:51.603164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:51.603257 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:51.603386 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:51.604932 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.606103 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.607844 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.608614 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.608780 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:51.612664 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:51.612718 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:51.612722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:51.612755 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:51.615684 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:51.616842 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:51.616846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:51.616849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:51.616992 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:51.617569 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:51.617574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:51.617576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:51.618030 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:51.618167 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:51.619298 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0807 06:48:51.619550 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":267,"threads_started":1}
I20260807 06:48:51.621407 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:51.600067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42721 (local address 127.27.54.66:40339)
0807 06:48:51.600943 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:51.601001 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:51.602879 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:51.603583 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:51.603591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:51.604151 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:51.604738 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:51.604755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.606261 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.606265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:51.607666 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:51.607679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:51.608857 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:51.608862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:51.609057 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:51.609834 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:51.609867 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:51.612367 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:51.615842 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:51.615854 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:51.615875 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:51.616690 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:51.617116 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:51.617124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:51.617130 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:51.617405 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:51.618204 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:51.618218 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:51.618775 (+ 557us) client_negotiation.cc:769] Sending connection context
0807 06:48:51.619122 (+ 347us) client_negotiation.cc:240] Negotiation successful
0807 06:48:51.619778 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":388,"threads_started":1}
I20260807 06:48:51.623009 1931 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42721
I20260807 06:48:51.623404 1931 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:51.624293 1931 heartbeater.cc:511] Master 127.27.54.126:42721 requested a full tablet report, sending...
I20260807 06:48:51.626070 1476 ts_manager.cc:194] Registered new tserver with Master: ef30e8eb68f04a13867f55d939bab758 (127.27.54.66:40621)
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
I20260807 06:48:51.627029 1476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:40339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:48:51.662398 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42721
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:48:52.011292 1940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:52.011695 1940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:52.011893 1940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:52.019989 1940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:52.020171 1940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:52.020378 1940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:52.030884 1940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:52.032126 1940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:52.033424 1940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:52.047581 1948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:52.047648 1949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:52.047704 1951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:52.048112 1940 server_base.cc:1034] running on GCE node
I20260807 06:48:52.049063 1940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:52.050012 1940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:52.051224 1940 hybrid_clock.cc:648] HybridClock initialized: now 1786085332051165 us; error 46 us; skew 500 ppm
I20260807 06:48:52.051586 1940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:48:52 dist-test-slave-3kk6 krb5kdc[1409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:48:52.055035 1940 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:48:52.056293 1940 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:52.056444 1940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:52.056787 1940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:52.060134 1940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "8adfb4efbe764ff084e0cd9ae548b9b5"
format_stamp: "Formatted at 2026-08-07 06:48:52 on dist-test-slave-3kk6"
server_key: "6e2f93081ddaf9214a55e9750cb1c9a7"
server_key_iv: "0e3287ee359ed0cc8351b5a24295214c"
server_key_version: "encryptionkey@0"
I20260807 06:48:52.061059 1940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "8adfb4efbe764ff084e0cd9ae548b9b5"
format_stamp: "Formatted at 2026-08-07 06:48:52 on dist-test-slave-3kk6"
server_key: "6e2f93081ddaf9214a55e9750cb1c9a7"
server_key_iv: "0e3287ee359ed0cc8351b5a24295214c"
server_key_version: "encryptionkey@0"
I20260807 06:48:52.061321 1740 heartbeater.cc:503] Master 127.27.54.126:42721 was elected leader, sending a full tablet report...
I20260807 06:48:52.066548 1940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 06:48:52.070174 1963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:52.071547 1940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:52.071769 1940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "8adfb4efbe764ff084e0cd9ae548b9b5"
format_stamp: "Formatted at 2026-08-07 06:48:52 on dist-test-slave-3kk6"
server_key: "6e2f93081ddaf9214a55e9750cb1c9a7"
server_key_iv: "0e3287ee359ed0cc8351b5a24295214c"
server_key_version: "encryptionkey@0"
I20260807 06:48:52.072017 1940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:52.110415 1940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:52.115253 1940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:52.115556 1940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:52.116634 1940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:52.118187 1940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:52.118301 1940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:52.118434 1940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:52.118521 1940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:52.153564 1940 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:43397
I20260807 06:48:52.153620 2119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:43397 every 8 connection(s)
I20260807 06:48:52.154527 1940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:52.160040 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 1940
I20260807 06:48:52.160230 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:52.160799 27865 external_mini_cluster.cc:1428] Setting key 4405b92237f0d30b607fc35f269be38d
Aug 07 06:48:52 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:48:52.175653 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.157347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:40449 (local address 127.27.54.126:42721)
0807 06:48:52.157771 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.157775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.158878 (+ 1103us) server_negotiation.cc:407] Connection header received
0807 06:48:52.161215 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.161221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.161335 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.161460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:52.163157 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.163968 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.165587 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.166040 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.166183 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.169728 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:52.169733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:52.169736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:52.169769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:52.172365 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.173124 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.173128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.173131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.173260 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.173717 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.173721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.173723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.174002 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:52.174174 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.174964 (+ 790us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.175207 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":203,"threads_started":1}
I20260807 06:48:52.176730 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.157945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42721 (local address 127.27.54.67:40449)
0807 06:48:52.158639 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:52.158750 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:52.160905 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:52.161711 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:52.161720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:52.162261 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:52.162956 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.162976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.164136 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.164141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.165426 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.165439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.166288 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.166296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.166532 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.167252 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:48:52.167275 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:52.169499 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:52.172508 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:52.172516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:52.172534 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:52.172983 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:52.173385 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:48:52.173390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:48:52.173393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:48:52.173584 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:48:52.174146 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:52.174155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:52.174507 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 06:48:52.174842 (+ 335us) client_negotiation.cc:240] Negotiation successful
0807 06:48:52.175418 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":274,"threads_started":1}
I20260807 06:48:52.178128 2120 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42721
I20260807 06:48:52.178478 2120 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:52.179200 2120 heartbeater.cc:511] Master 127.27.54.126:42721 requested a full tablet report, sending...
I20260807 06:48:52.180651 1476 ts_manager.cc:194] Registered new tserver with Master: 8adfb4efbe764ff084e0cd9ae548b9b5 (127.27.54.67:43397)
I20260807 06:48:52.181443 1476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:40449
I20260807 06:48:52.188674 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:48:52.206562 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.193525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.27.54.126:42721)
0807 06:48:52.193692 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.193696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.194290 (+ 594us) server_negotiation.cc:407] Connection header received
0807 06:48:52.194446 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.194450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.194513 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.194598 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:52.196118 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.196827 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.198081 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.198486 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.198585 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.200997 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:52.201002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:52.201005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:52.201039 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:52.203968 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.204669 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.204674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.204676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.204820 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.205233 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.205236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.205238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.205566 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:52.205760 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.206132 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.206323 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:48:52.211261 1476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:52.218552 1476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:52.257148 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.232460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.27.54.66:40621)
0807 06:48:52.244830 (+ 12370us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.244844 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.246574 (+ 1730us) server_negotiation.cc:407] Connection header received
0807 06:48:52.246732 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.246741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.247005 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.247218 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:52.249009 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.250636 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.254160 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.255770 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.255945 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.256172 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.256566 (+ 394us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.256758 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12102,"thread_start_us":13289,"threads_started":1}
I20260807 06:48:52.257203 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.247344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.27.54.67:43397)
0807 06:48:52.248166 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.248175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.249197 (+ 1022us) server_negotiation.cc:407] Connection header received
0807 06:48:52.249540 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.249549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.249778 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.249982 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:52.251017 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.252262 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.254399 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.255825 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.255944 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.256190 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.256654 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.256838 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":421,"threads_started":1}
I20260807 06:48:52.257274 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.247049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:43397 (local address 127.0.0.1:34816)
0807 06:48:52.249059 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:52.249078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:52.249206 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:52.250264 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:52.250268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:52.250293 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:52.250719 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.250734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.252425 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.252432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.254196 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.254212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.256334 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.256338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.256459 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.256481 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:48:52.256594 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 06:48:52.256825 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1885,"mutex_wait_us":94,"thread_start_us":1257,"threads_started":1}
I20260807 06:48:52.258127 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.231845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:40621 (local address 127.0.0.1:54328)
0807 06:48:52.246429 (+ 14584us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:52.246444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:52.246551 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:52.247512 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:52.247520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:52.247562 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:52.248825 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.248842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.250825 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.250829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.253967 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.253976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.256020 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.256027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.256236 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.256303 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 06:48:52.257526 (+ 1223us) client_negotiation.cc:240] Negotiation successful
0807 06:48:52.257660 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14484,"mutex_wait_us":691,"spinlock_wait_cycles":384,"thread_start_us":13224,"threads_started":1}
I20260807 06:48:52.258154 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.231590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44575 (local address 127.0.0.1:50322)
0807 06:48:52.245470 (+ 13880us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:52.245517 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:52.245697 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:52.247512 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:52.247520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:52.247569 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:52.248551 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.248582 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.250736 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.250745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.253959 (+ 3214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.253976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.256532 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.256538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.256725 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.256756 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:48:52.256842 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 06:48:52.257000 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13623,"mutex_wait_us":330,"spinlock_wait_cycles":3840,"thread_start_us":14986,"threads_started":1}
I20260807 06:48:52.261554 1841 tablet_service.cc:1467] Processing CreateTablet for tablet cad4df2106d14a84814aeaaa28f941f7 (DEFAULT_TABLE table=test-table [id=c137a479c120411689d920e68d479342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:52.261569 2033 tablet_service.cc:1467] Processing CreateTablet for tablet cad4df2106d14a84814aeaaa28f941f7 (DEFAULT_TABLE table=test-table [id=c137a479c120411689d920e68d479342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:52.262941 1841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad4df2106d14a84814aeaaa28f941f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:52.262940 2033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad4df2106d14a84814aeaaa28f941f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:52.263173 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.231806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.27.54.65:44575)
0807 06:48:52.246602 (+ 14796us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.246611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.246637 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:48:52.246742 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.246749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.247007 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.247218 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:52.248873 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.250537 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.254228 (+ 3691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.255991 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.256140 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.256373 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.257775 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.257974 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14590,"thread_start_us":14384,"threads_started":1}
I20260807 06:48:52.267117 1651 tablet_service.cc:1467] Processing CreateTablet for tablet cad4df2106d14a84814aeaaa28f941f7 (DEFAULT_TABLE table=test-table [id=c137a479c120411689d920e68d479342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:52.268807 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cad4df2106d14a84814aeaaa28f941f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:52.274879 2149 tablet_bootstrap.cc:492] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Bootstrap starting.
I20260807 06:48:52.279415 2149 tablet_bootstrap.cc:654] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:52.280519 2149 log.cc:824] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:52.283607 2149 tablet_bootstrap.cc:492] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: No bootstrap required, opened a new log
I20260807 06:48:52.283903 2149 ts_tablet_manager.cc:1397] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 06:48:52.285054 2153 tablet_bootstrap.cc:492] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Bootstrap starting.
I20260807 06:48:52.286888 2150 tablet_bootstrap.cc:492] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Bootstrap starting.
I20260807 06:48:52.288951 2153 tablet_bootstrap.cc:654] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:52.290416 2153 log.cc:824] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:52.290870 2150 tablet_bootstrap.cc:654] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:52.292173 2150 log.cc:824] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:52.293871 2149 raft_consensus.cc:348] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.294260 2149 raft_consensus.cc:374] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:52.294378 2149 raft_consensus.cc:729] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef30e8eb68f04a13867f55d939bab758, State: Initialized, Role: FOLLOWER
I20260807 06:48:52.294396 2153 tablet_bootstrap.cc:492] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: No bootstrap required, opened a new log
I20260807 06:48:52.294771 2153 ts_tablet_manager.cc:1397] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260807 06:48:52.295061 2149 consensus_queue.cc:260] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.296360 1931 heartbeater.cc:503] Master 127.27.54.126:42721 was elected leader, sending a full tablet report...
I20260807 06:48:52.296783 2149 ts_tablet_manager.cc:1428] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260807 06:48:52.302055 2150 tablet_bootstrap.cc:492] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: No bootstrap required, opened a new log
I20260807 06:48:52.302392 2150 ts_tablet_manager.cc:1397] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.009s
I20260807 06:48:52.304898 2153 raft_consensus.cc:348] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.305326 2153 raft_consensus.cc:374] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:52.305441 2153 raft_consensus.cc:729] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376a43d564b24c7d9f28e7da2e43fe9f, State: Initialized, Role: FOLLOWER
I20260807 06:48:52.306066 2153 consensus_queue.cc:260] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.307889 2153 ts_tablet_manager.cc:1428] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260807 06:48:52.312256 2150 raft_consensus.cc:348] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.312656 2150 raft_consensus.cc:374] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:52.312785 2150 raft_consensus.cc:729] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adfb4efbe764ff084e0cd9ae548b9b5, State: Initialized, Role: FOLLOWER
I20260807 06:48:52.313432 2150 consensus_queue.cc:260] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.314569 2120 heartbeater.cc:503] Master 127.27.54.126:42721 was elected leader, sending a full tablet report...
I20260807 06:48:52.315106 2150 ts_tablet_manager.cc:1428] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
W20260807 06:48:52.348658 1932 tablet.cc:2369] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:52.350772 2158 raft_consensus.cc:482] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:52.351348 2158 raft_consensus.cc:504] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.353253 2158 leader_election.cc:288] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef30e8eb68f04a13867f55d939bab758 (127.27.54.66:40621), 8adfb4efbe764ff084e0cd9ae548b9b5 (127.27.54.67:43397)
I20260807 06:48:52.362604 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.353893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:41629 (local address 127.27.54.67:43397)
0807 06:48:52.354142 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.354148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.355381 (+ 1233us) server_negotiation.cc:407] Connection header received
0807 06:48:52.355539 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.355545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.355624 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.355748 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:52.356427 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.357471 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.360150 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.361463 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.361642 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.361718 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.362187 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.362327 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:48:52.362778 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.353884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:43397 (local address 127.27.54.65:41629)
0807 06:48:52.355272 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:52.355286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:52.355386 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:52.355820 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:52.355826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:52.355859 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:52.356235 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.356245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.357604 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.357611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.359937 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.359963 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.361734 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.361741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.361990 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.362021 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:48:52.362160 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 06:48:52.362312 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":375,"thread_start_us":655,"threads_started":1}
I20260807 06:48:52.362854 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.353893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:53011 (local address 127.27.54.66:40621)
0807 06:48:52.354098 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.354103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.355001 (+ 898us) server_negotiation.cc:407] Connection header received
0807 06:48:52.355253 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.355257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.355323 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.355482 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:48:52.356105 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.357239 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.360150 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.361474 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.361641 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.361718 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.362391 (+ 673us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.362546 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:48:52.363433 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.353884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:40621 (local address 127.27.54.65:53011)
0807 06:48:52.354837 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:52.354863 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:52.355083 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:52.355501 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:52.355506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:52.355537 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:48:52.355928 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.355940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.357395 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.357399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.359927 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:52.359956 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.361939 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.361947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:52.362181 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.362215 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:48:52.362912 (+ 697us) client_negotiation.cc:240] Negotiation successful
0807 06:48:52.363069 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":10,"thread_start_us":481,"threads_started":1}
I20260807 06:48:52.364269 2058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad4df2106d14a84814aeaaa28f941f7" candidate_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" is_pre_election: true
I20260807 06:48:52.364471 1872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad4df2106d14a84814aeaaa28f941f7" candidate_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef30e8eb68f04a13867f55d939bab758" is_pre_election: true
I20260807 06:48:52.364850 2058 raft_consensus.cc:2454] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376a43d564b24c7d9f28e7da2e43fe9f in term 0.
I20260807 06:48:52.365022 1872 raft_consensus.cc:2454] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376a43d564b24c7d9f28e7da2e43fe9f in term 0.
I20260807 06:48:52.365800 1579 leader_election.cc:302] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376a43d564b24c7d9f28e7da2e43fe9f, 8adfb4efbe764ff084e0cd9ae548b9b5; no voters:
I20260807 06:48:52.366287 2158 raft_consensus.cc:2781] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:52.366477 2158 raft_consensus.cc:482] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:52.366570 2158 raft_consensus.cc:3037] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:52.369880 2158 raft_consensus.cc:504] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.370684 2158 leader_election.cc:288] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [CANDIDATE]: Term 1 election: Requested vote from peers ef30e8eb68f04a13867f55d939bab758 (127.27.54.66:40621), 8adfb4efbe764ff084e0cd9ae548b9b5 (127.27.54.67:43397)
I20260807 06:48:52.371294 2058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad4df2106d14a84814aeaaa28f941f7" candidate_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5"
I20260807 06:48:52.371330 1872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad4df2106d14a84814aeaaa28f941f7" candidate_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef30e8eb68f04a13867f55d939bab758"
I20260807 06:48:52.371546 2058 raft_consensus.cc:3037] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:52.371558 1872 raft_consensus.cc:3037] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:52.373997 2058 raft_consensus.cc:2454] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376a43d564b24c7d9f28e7da2e43fe9f in term 1.
I20260807 06:48:52.374590 1579 leader_election.cc:302] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376a43d564b24c7d9f28e7da2e43fe9f, 8adfb4efbe764ff084e0cd9ae548b9b5; no voters:
I20260807 06:48:52.374965 2158 raft_consensus.cc:2781] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:52.374958 1872 raft_consensus.cc:2454] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376a43d564b24c7d9f28e7da2e43fe9f in term 1.
I20260807 06:48:52.375746 2158 raft_consensus.cc:686] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 1 LEADER]: Becoming Leader. State: Replica: 376a43d564b24c7d9f28e7da2e43fe9f, State: Running, Role: LEADER
I20260807 06:48:52.376371 2158 consensus_queue.cc:237] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } }
I20260807 06:48:52.381455 1475 catalog_manager.cc:5495] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 376a43d564b24c7d9f28e7da2e43fe9f (127.27.54.65). New cstate: current_term: 1 leader_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376a43d564b24c7d9f28e7da2e43fe9f" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 } health_report { overall_health: UNKNOWN } } }
W20260807 06:48:52.406291 2121 tablet.cc:2369] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:48:52.406595 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.400754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.27.54.65:44575)
0807 06:48:52.400968 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.400973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.401077 (+ 104us) server_negotiation.cc:407] Connection header received
0807 06:48:52.401270 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.401274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.401360 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.401502 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:48:52.401894 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.402622 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.404193 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.404625 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.404730 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.405283 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:48:52.405898 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:48:52.406026 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.406236 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.406319 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 06:48:52.416077 1872 raft_consensus.cc:1261] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 1 FOLLOWER]: Refusing update from remote peer 376a43d564b24c7d9f28e7da2e43fe9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:52.416077 2058 raft_consensus.cc:1261] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 1 FOLLOWER]: Refusing update from remote peer 376a43d564b24c7d9f28e7da2e43fe9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:52.417248 2163 consensus_queue.cc:1035] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef30e8eb68f04a13867f55d939bab758" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:52.417712 2158 consensus_queue.cc:1035] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8adfb4efbe764ff084e0cd9ae548b9b5" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:52.434134 2170 mvcc.cc:205] Tried to move back new op lower bound from 7315805521563926528 to 7315805521418055680. Current Snapshot: MvccSnapshot[applied={T|T < 7315805521563926528}]
I20260807 06:48:52.445748 1475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:52.446055 1475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:52.449913 1475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c0fa9a49d6d4250a30d9b3911968b66: Sending DeleteTablet for 3 replicas of tablet cad4df2106d14a84814aeaaa28f941f7
I20260807 06:48:52.451432 1841 tablet_service.cc:1514] Processing DeleteTablet for tablet cad4df2106d14a84814aeaaa28f941f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:52 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:54328
I20260807 06:48:52.451438 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet cad4df2106d14a84814aeaaa28f941f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:52 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:50322
I20260807 06:48:52.451960 2033 tablet_service.cc:1514] Processing DeleteTablet for tablet cad4df2106d14a84814aeaaa28f941f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:52 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:34816
I20260807 06:48:52.453987 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 1553
I20260807 06:48:52.463363 2182 tablet_replica.cc:330] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: stopping tablet replica
I20260807 06:48:52.463707 2187 tablet_replica.cc:330] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: stopping tablet replica
I20260807 06:48:52.464381 2187 raft_consensus.cc:2229] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:52.465004 2182 raft_consensus.cc:2229] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:52.465086 2187 raft_consensus.cc:2258] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:52.466111 2182 raft_consensus.cc:2258] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:52.467840 2186 tablet_replica.cc:330] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: stopping tablet replica
I20260807 06:48:52.468909 2187 ts_tablet_manager.cc:1905] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:52.469162 2182 ts_tablet_manager.cc:1905] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:52.469542 2186 raft_consensus.cc:2229] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:52.471243 2186 raft_consensus.cc:2258] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 06:48:52 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:48:52.477193 2182 ts_tablet_manager.cc:1918] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:52.477415 2182 log.cc:1196] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/cad4df2106d14a84814aeaaa28f941f7
I20260807 06:48:52.477420 2186 ts_tablet_manager.cc:1905] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:52.478012 2182 ts_tablet_manager.cc:1939] T cad4df2106d14a84814aeaaa28f941f7 P ef30e8eb68f04a13867f55d939bab758: Deleting consensus metadata
I20260807 06:48:52.478204 2187 ts_tablet_manager.cc:1918] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:52.478461 2187 log.cc:1196] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/cad4df2106d14a84814aeaaa28f941f7
I20260807 06:48:52.479135 2187 ts_tablet_manager.cc:1939] T cad4df2106d14a84814aeaaa28f941f7 P 8adfb4efbe764ff084e0cd9ae548b9b5: Deleting consensus metadata
I20260807 06:48:52.480216 1448 catalog_manager.cc:4841] TS ef30e8eb68f04a13867f55d939bab758 (127.27.54.66:40621): tablet cad4df2106d14a84814aeaaa28f941f7 (table test-table [id=c137a479c120411689d920e68d479342]) successfully deleted
I20260807 06:48:52.480998 1451 catalog_manager.cc:4841] TS 8adfb4efbe764ff084e0cd9ae548b9b5 (127.27.54.67:43397): tablet cad4df2106d14a84814aeaaa28f941f7 (table test-table [id=c137a479c120411689d920e68d479342]) successfully deleted
I20260807 06:48:52.485312 2186 ts_tablet_manager.cc:1918] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:52.485512 2186 log.cc:1196] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/cad4df2106d14a84814aeaaa28f941f7
I20260807 06:48:52.486125 2186 ts_tablet_manager.cc:1939] T cad4df2106d14a84814aeaaa28f941f7 P 376a43d564b24c7d9f28e7da2e43fe9f: Deleting consensus metadata
I20260807 06:48:52.488188 1451 catalog_manager.cc:4841] TS 376a43d564b24c7d9f28e7da2e43fe9f (127.27.54.65:44575): tablet cad4df2106d14a84814aeaaa28f941f7 (table test-table [id=c137a479c120411689d920e68d479342]) successfully deleted
I20260807 06:48:52.489521 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.455559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.27.54.65:44575)
0807 06:48:52.455777 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.455784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.460193 (+ 4409us) server_negotiation.cc:407] Connection header received
0807 06:48:52.460455 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.460462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.460566 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.460706 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:52.465354 (+ 4648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.466706 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.468728 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.469496 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.469614 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.482203 (+ 12589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:52.482215 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:52.482229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:52.482275 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:52.486053 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.487025 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.487035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.487037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.487198 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.487713 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.487719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.487722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.488213 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:52.488523 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.489004 (+ 481us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.489200 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 06:48:52.490450 1732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:52.593369 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 1553
I20260807 06:48:52.632241 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 1747
Aug 07 06:48:52 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:48:52.650168 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.633192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.27.54.66:40621)
0807 06:48:52.633373 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.633378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.633397 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:48:52.633545 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.633550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.633613 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.633736 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:52.635507 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.636351 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.638108 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.638528 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.638625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.643878 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:52.643891 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:52.643906 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:52.643950 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:52.647183 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.647961 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.647968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.647970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.648129 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.648498 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.648502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.648504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.648978 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:52.649205 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.649495 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.649806 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:48:52.650965 1926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:52.744702 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 1747
I20260807 06:48:52.771960 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 1940
Aug 07 06:48:52 dist-test-slave-3kk6 krb5kdc[1409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:48:52.788352 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:52.772899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.27.54.67:43397)
0807 06:48:52.773109 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:52.773117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:52.773140 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:48:52.773380 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:52.773385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:52.773468 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:52.773584 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:52.775028 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.775884 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.777253 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:52.777679 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:52.777833 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:52.782492 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:52.782510 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:52.782527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:48:52.782567 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:52.785423 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.786141 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.786149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.786152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.786294 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:48:52.786729 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:48:52.786736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:48:52.786743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:48:52.787114 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:52.787437 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:52.787764 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 06:48:52.788012 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 06:48:52.789036 2114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:52.880287 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 1940
I20260807 06:48:52.902434 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 1428
I20260807 06:48:52.903692 1513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:53.016414 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 1428
2026-08-07T06:48:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4709 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T06:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:53Z Disabled control of system clock
I20260807 06:48:53.106009 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34547
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 06:48:53.458678 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:53.459007 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:53.459197 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:53.466755 2201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:53.466890 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:53.466992 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:53.467082 2201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:53.467134 2201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:48:53.476854 2201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:34547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:53.478310 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:53.480247 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:53.493248 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:53.493291 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:53.493317 2215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:53.494216 2201 server_base.cc:1034] running on GCE node
I20260807 06:48:53.495183 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:53.496722 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:53.497963 2201 hybrid_clock.cc:648] HybridClock initialized: now 1786085333497909 us; error 43 us; skew 500 ppm
I20260807 06:48:53.498410 2201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:53.500993 2201 webserver.cc:460] Webserver started at http://127.27.54.126:42129/ using document root <none> and password file <none>
I20260807 06:48:53.501780 2201 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:53.501909 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:53.502275 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:53.505452 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "269fe4638cf340478008780c449126e6"
format_stamp: "Formatted at 2026-08-07 06:48:53 on dist-test-slave-3kk6"
server_key: "d916bcdd09d81ad8a64e3324c1cc0d4d"
server_key_iv: "e7d1208e4e0c4792e741a2efe2c64f41"
server_key_version: "encryptionkey@0"
I20260807 06:48:53.506366 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "269fe4638cf340478008780c449126e6"
format_stamp: "Formatted at 2026-08-07 06:48:53 on dist-test-slave-3kk6"
server_key: "d916bcdd09d81ad8a64e3324c1cc0d4d"
server_key_iv: "e7d1208e4e0c4792e741a2efe2c64f41"
server_key_version: "encryptionkey@0"
I20260807 06:48:53.511962 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:48:53.515586 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:53.516927 2201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:53.517122 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "269fe4638cf340478008780c449126e6"
format_stamp: "Formatted at 2026-08-07 06:48:53 on dist-test-slave-3kk6"
server_key: "d916bcdd09d81ad8a64e3324c1cc0d4d"
server_key_iv: "e7d1208e4e0c4792e741a2efe2c64f41"
server_key_version: "encryptionkey@0"
I20260807 06:48:53.517390 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:53.523587 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:53.524418 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:53.524740 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:53.544011 2201 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:34547
I20260807 06:48:53.544049 2306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:34547 every 8 connection(s)
I20260807 06:48:53.545486 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:53.549392 2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:53.552822 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 2201
I20260807 06:48:53.553079 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:53.553912 27865 external_mini_cluster.cc:1428] Setting key f33c96f723f230f28c64190eebe62767
I20260807 06:48:53.562731 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:53.556171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.27.54.126:34547)
0807 06:48:53.557213 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:53.557224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:53.557266 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:48:53.558993 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:53.559017 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:53.559466 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:53.559812 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:53.561040 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:53.561047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:53.561059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:53.561074 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:53.561138 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:53.561163 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:53.561346 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:53.561494 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 06:48:53.561760 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":395,"threads_started":1}
I20260807 06:48:53.563197 2307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6: Bootstrap starting.
I20260807 06:48:53.566823 2307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:53.567838 2307 log.cc:824] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:53.570528 2307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6: No bootstrap required, opened a new log
I20260807 06:48:53.578959 2307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269fe4638cf340478008780c449126e6" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34547 } }
I20260807 06:48:53.579506 2307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:53.579638 2307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269fe4638cf340478008780c449126e6, State: Initialized, Role: FOLLOWER
I20260807 06:48:53.580255 2307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269fe4638cf340478008780c449126e6" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34547 } }
I20260807 06:48:53.580523 2307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:53.580674 2307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:53.580862 2307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:53.583660 2307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269fe4638cf340478008780c449126e6" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34547 } }
I20260807 06:48:53.584262 2307 leader_election.cc:302] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 269fe4638cf340478008780c449126e6; no voters:
I20260807 06:48:53.584977 2307 leader_election.cc:288] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:53.585186 2318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:53.586021 2318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [term 1 LEADER]: Becoming Leader. State: Replica: 269fe4638cf340478008780c449126e6, State: Running, Role: LEADER
I20260807 06:48:53.586740 2318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269fe4638cf340478008780c449126e6" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34547 } }
I20260807 06:48:53.587349 2307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:53.589891 2319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "269fe4638cf340478008780c449126e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269fe4638cf340478008780c449126e6" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34547 } } }
I20260807 06:48:53.590418 2319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:53.592001 2330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:53.591936 2321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 269fe4638cf340478008780c449126e6. Latest consensus state: current_term: 1 leader_uuid: "269fe4638cf340478008780c449126e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269fe4638cf340478008780c449126e6" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34547 } } }
I20260807 06:48:53.595110 2321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:53.600514 2330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:53.609308 2330 catalog_manager.cc:1347] Generated new cluster ID: f360c2919b5d4cc0a01d263d2d895e75
I20260807 06:48:53.609443 2330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:53.648756 2330 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:53.649240 2330 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:53.670601 2330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6: Generated new TSK 0
I20260807 06:48:53.671602 2330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:48:53.681190 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34547
--builtin_ntp_servers=127.27.54.84:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 06:48:54.049081 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:54.049400 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:54.049490 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:54.057060 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:54.057183 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:54.057296 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:48:54.066851 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:54.068250 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:54.070082 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:54.083740 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:54.083853 2350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:54.083850 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:54.085104 2343 server_base.cc:1034] running on GCE node
I20260807 06:48:54.085757 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:54.086862 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:54.088104 2343 hybrid_clock.cc:648] HybridClock initialized: now 1786085334088056 us; error 42 us; skew 500 ppm
I20260807 06:48:54.088464 2343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:54.091058 2343 webserver.cc:460] Webserver started at http://127.27.54.65:36879/ using document root <none> and password file <none>
I20260807 06:48:54.091889 2343 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:54.091988 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:54.092408 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:54.095943 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "d2121892450e4941ac8edf62dbe950d7"
format_stamp: "Formatted at 2026-08-07 06:48:54 on dist-test-slave-3kk6"
server_key: "7e38d6aab95d416b3bddd40cd30adb2a"
server_key_iv: "9f0f58ca89662ccdc391e17229a9f25e"
server_key_version: "encryptionkey@0"
I20260807 06:48:54.097246 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "d2121892450e4941ac8edf62dbe950d7"
format_stamp: "Formatted at 2026-08-07 06:48:54 on dist-test-slave-3kk6"
server_key: "7e38d6aab95d416b3bddd40cd30adb2a"
server_key_iv: "9f0f58ca89662ccdc391e17229a9f25e"
server_key_version: "encryptionkey@0"
I20260807 06:48:54.105844 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.007s
I20260807 06:48:54.110085 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:54.111688 2343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:48:54.111896 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "d2121892450e4941ac8edf62dbe950d7"
format_stamp: "Formatted at 2026-08-07 06:48:54 on dist-test-slave-3kk6"
server_key: "7e38d6aab95d416b3bddd40cd30adb2a"
server_key_iv: "9f0f58ca89662ccdc391e17229a9f25e"
server_key_version: "encryptionkey@0"
I20260807 06:48:54.112180 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:54.118281 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:54.119262 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:54.119575 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:54.120605 2343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:54.122351 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:54.122475 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:54.122610 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:54.122725 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 06:48:54.158253 2343 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:34527
I20260807 06:48:54.158300 2517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:34527 every 8 connection(s)
I20260807 06:48:54.159663 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:48:54.163039 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 2343
I20260807 06:48:54.163285 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:48:54.163959 27865 external_mini_cluster.cc:1428] Setting key 5412fc8093776b4111f7fe26f920f100
I20260807 06:48:54.166975 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34547
--builtin_ntp_servers=127.27.54.84:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 06:48:54.169207 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:54.162443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:35697 (local address 127.27.54.126:34547)
0807 06:48:54.163012 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:54.163022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:54.164625 (+ 1603us) server_negotiation.cc:407] Connection header received
0807 06:48:54.166623 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:54.166628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:54.166728 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:54.166861 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:54.168385 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:54.168391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:54.168395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:54.168416 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:54.168470 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:54.168482 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:54.168620 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:54.168833 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 06:48:54.168907 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":282,"threads_started":1}
I20260807 06:48:54.170459 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:54.163128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34547 (local address 127.27.54.65:35697)
0807 06:48:54.164403 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:54.164459 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:54.166335 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:54.167144 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:54.167155 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:54.167285 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:54.168045 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:54.168047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:54.168088 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:54.168181 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:54.168584 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:54.168598 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:54.168606 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:48:54.168931 (+ 325us) client_negotiation.cc:240] Negotiation successful
0807 06:48:54.169217 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":476,"threads_started":1}
I20260807 06:48:54.172401 2518 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34547
I20260807 06:48:54.172734 2518 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:54.173441 2518 heartbeater.cc:511] Master 127.27.54.126:34547 requested a full tablet report, sending...
I20260807 06:48:54.176009 2252 ts_manager.cc:194] Registered new tserver with Master: d2121892450e4941ac8edf62dbe950d7 (127.27.54.65:34527)
W20260807 06:48:54.520678 2522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:54.521003 2522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:54.521093 2522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:54.528641 2522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:54.528767 2522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:54.528879 2522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:48:54.538441 2522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:54.540107 2522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:54.542014 2522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:54.559737 2532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:54.559566 2535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:54.560024 2530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:54.560806 2522 server_base.cc:1034] running on GCE node
I20260807 06:48:54.561528 2522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:54.562572 2522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:54.563833 2522 hybrid_clock.cc:648] HybridClock initialized: now 1786085334563785 us; error 38 us; skew 500 ppm
I20260807 06:48:54.564167 2522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:54.566820 2522 webserver.cc:460] Webserver started at http://127.27.54.66:34213/ using document root <none> and password file <none>
I20260807 06:48:54.567554 2522 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:54.567651 2522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:54.567961 2522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:54.571080 2522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "36374125e1d448cd821c1f14c7dab9b7"
format_stamp: "Formatted at 2026-08-07 06:48:54 on dist-test-slave-3kk6"
server_key: "3bd568c9b07df70092f690c61482ddc7"
server_key_iv: "7d37b1d24754b205119b40661fcb397a"
server_key_version: "encryptionkey@0"
I20260807 06:48:54.571938 2522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "36374125e1d448cd821c1f14c7dab9b7"
format_stamp: "Formatted at 2026-08-07 06:48:54 on dist-test-slave-3kk6"
server_key: "3bd568c9b07df70092f690c61482ddc7"
server_key_iv: "7d37b1d24754b205119b40661fcb397a"
server_key_version: "encryptionkey@0"
I20260807 06:48:54.577527 2522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:48:54.581175 2548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:54.582561 2522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:48:54.582805 2522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "36374125e1d448cd821c1f14c7dab9b7"
format_stamp: "Formatted at 2026-08-07 06:48:54 on dist-test-slave-3kk6"
server_key: "3bd568c9b07df70092f690c61482ddc7"
server_key_iv: "7d37b1d24754b205119b40661fcb397a"
server_key_version: "encryptionkey@0"
I20260807 06:48:54.583024 2522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:54.589498 2522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:54.590456 2522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:54.590802 2522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:54.591789 2522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:54.593505 2522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:54.593595 2522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:54.593683 2522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:54.593737 2522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:54.629196 2522 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:41277
I20260807 06:48:54.629259 2710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:41277 every 8 connection(s)
I20260807 06:48:54.630604 2522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:48:54.635952 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 2522
I20260807 06:48:54.636168 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:48:54.636752 27865 external_mini_cluster.cc:1428] Setting key 11ff42e39a57dd2ab8dcbaec3ea8f7ed
I20260807 06:48:54.639637 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:54.633276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:46129 (local address 127.27.54.126:34547)
0807 06:48:54.633472 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:54.633476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:54.634878 (+ 1402us) server_negotiation.cc:407] Connection header received
0807 06:48:54.637223 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:54.637227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:54.637286 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:54.637382 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:54.638851 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:54.638857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:54.638862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:54.638883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:54.638927 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:54.638940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:54.639032 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:54.639292 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 06:48:54.639356 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:48:54.640141 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34547
--builtin_ntp_servers=127.27.54.84:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 06:48:54.640630 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:54.633759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34547 (local address 127.27.54.66:46129)
0807 06:48:54.634688 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:54.634746 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:54.636894 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:54.637628 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:54.637636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:54.637725 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:54.638453 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:54.638457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:54.638506 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:54.638619 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:54.639045 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:54.639058 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:54.639066 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:48:54.639326 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 06:48:54.639527 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":273,"threads_started":1}
I20260807 06:48:54.642158 2711 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34547
I20260807 06:48:54.642480 2711 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:54.643119 2711 heartbeater.cc:511] Master 127.27.54.126:34547 requested a full tablet report, sending...
I20260807 06:48:54.644541 2252 ts_manager.cc:194] Registered new tserver with Master: 36374125e1d448cd821c1f14c7dab9b7 (127.27.54.66:41277)
W20260807 06:48:54.991247 2715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:54.991581 2715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:54.991792 2715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:54.999390 2715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:54.999514 2715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:54.999696 2715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:48:55.009164 2715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:55.010627 2715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:55.012547 2715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:55.026492 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:55.026420 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:55.026685 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:55.027705 2715 server_base.cc:1034] running on GCE node
I20260807 06:48:55.028596 2715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:55.030076 2715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:55.031311 2715 hybrid_clock.cc:648] HybridClock initialized: now 1786085335031237 us; error 60 us; skew 500 ppm
I20260807 06:48:55.031780 2715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:55.037223 2715 webserver.cc:460] Webserver started at http://127.27.54.67:35549/ using document root <none> and password file <none>
I20260807 06:48:55.038020 2715 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:55.038161 2715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:55.038546 2715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:55.042546 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "3350edfa1640463d951abb1d412d70d4"
format_stamp: "Formatted at 2026-08-07 06:48:55 on dist-test-slave-3kk6"
server_key: "780b8a8949649eb32ae7668c6265d227"
server_key_iv: "1bd6dcbed4a3ae2eda98de1217d39562"
server_key_version: "encryptionkey@0"
I20260807 06:48:55.043538 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "3350edfa1640463d951abb1d412d70d4"
format_stamp: "Formatted at 2026-08-07 06:48:55 on dist-test-slave-3kk6"
server_key: "780b8a8949649eb32ae7668c6265d227"
server_key_iv: "1bd6dcbed4a3ae2eda98de1217d39562"
server_key_version: "encryptionkey@0"
I20260807 06:48:55.049250 2715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:48:55.053148 2733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:55.054504 2715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:48:55.054795 2715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "3350edfa1640463d951abb1d412d70d4"
format_stamp: "Formatted at 2026-08-07 06:48:55 on dist-test-slave-3kk6"
server_key: "780b8a8949649eb32ae7668c6265d227"
server_key_iv: "1bd6dcbed4a3ae2eda98de1217d39562"
server_key_version: "encryptionkey@0"
I20260807 06:48:55.055100 2715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:55.061347 2715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:55.062234 2715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:55.062532 2715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:55.063565 2715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:48:55.065054 2715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:48:55.065186 2715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:55.065327 2715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:48:55.065418 2715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:55.101567 2715 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:43229
I20260807 06:48:55.101663 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:43229 every 8 connection(s)
I20260807 06:48:55.103072 2715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:48:55.108706 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 2715
I20260807 06:48:55.108946 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:48:55.109400 27865 external_mini_cluster.cc:1428] Setting key 5221a0a3634eb49900cd4ca6484ff80d
I20260807 06:48:55.112532 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.105886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:40607 (local address 127.27.54.126:34547)
0807 06:48:55.106069 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.106074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.107757 (+ 1683us) server_negotiation.cc:407] Connection header received
0807 06:48:55.109742 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.109748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.109833 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.109972 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.111809 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.111813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.111817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.111833 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.111868 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.111880 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.111972 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.112238 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.112288 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:48:55.113813 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.106366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34547 (local address 127.27.54.67:40607)
0807 06:48:55.107562 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.107602 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:55.109395 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:55.110262 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:55.110273 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:55.110388 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:55.111417 (+ 1029us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:55.111420 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:55.111473 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:55.111577 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:55.111991 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:55.112007 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:55.112017 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 06:48:55.112357 (+ 340us) client_negotiation.cc:240] Negotiation successful
0807 06:48:55.112719 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":530,"threads_started":1}
I20260807 06:48:55.115363 2879 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34547
I20260807 06:48:55.115773 2879 heartbeater.cc:464] Registering TS with master...
I20260807 06:48:55.116487 2879 heartbeater.cc:511] Master 127.27.54.126:34547 requested a full tablet report, sending...
I20260807 06:48:55.118072 2252 ts_manager.cc:194] Registered new tserver with Master: 3350edfa1640463d951abb1d412d70d4 (127.27.54.67:43229)
I20260807 06:48:55.125710 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:48:55.133414 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.130500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.27.54.126:34547)
0807 06:48:55.130747 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.130756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.131297 (+ 541us) server_negotiation.cc:407] Connection header received
0807 06:48:55.131517 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.131523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.131594 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.131717 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.132739 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.132745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.132748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.132766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.132801 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.132813 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.132928 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.133023 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.133075 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:48:55.137642 2252 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:48:55.144608 2252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:48:55.179005 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.162948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.27.54.66:41277)
0807 06:48:55.167189 (+ 4241us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.167202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.174245 (+ 7043us) server_negotiation.cc:407] Connection header received
0807 06:48:55.174546 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.174556 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.174843 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.174980 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.177933 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.177943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.177958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.177990 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.178057 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.178081 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.178205 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.178553 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.178703 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3551,"thread_start_us":5969,"threads_started":1}
I20260807 06:48:55.179311 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.157341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:43229 (local address 127.0.0.1:49502)
0807 06:48:55.173422 (+ 16081us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.173490 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:55.173644 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:55.176170 (+ 2526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:55.176174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:55.176278 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:55.177790 (+ 1512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:55.177794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:55.177811 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:55.177856 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:55.178458 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:55.178463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:55.178468 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:48:55.178608 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 06:48:55.178734 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15792,"thread_start_us":15753,"threads_started":1}
I20260807 06:48:55.179464 2518 heartbeater.cc:503] Master 127.27.54.126:34547 was elected leader, sending a full tablet report...
I20260807 06:48:55.179857 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.157604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:41277 (local address 127.0.0.1:49394)
0807 06:48:55.174070 (+ 16466us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.174099 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:55.174357 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:55.176165 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:55.176172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:55.176278 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:55.177580 (+ 1302us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:55.177582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:55.177615 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:55.177715 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:55.178298 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:55.178309 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:55.178316 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:48:55.179398 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0807 06:48:55.179462 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16299,"thread_start_us":18950,"threads_started":1}
I20260807 06:48:55.180074 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.158068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.27.54.67:43229)
0807 06:48:55.174452 (+ 16384us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.174463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.174500 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:48:55.174574 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.174580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.174842 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.174995 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.178059 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.178072 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.178086 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.178110 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.178187 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.178215 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.178397 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.179555 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.179672 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16203,"thread_start_us":15996,"threads_started":1}
I20260807 06:48:55.181322 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.158903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.27.54.65:34527)
0807 06:48:55.166169 (+ 7266us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.166183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.177100 (+ 10917us) server_negotiation.cc:407] Connection header received
0807 06:48:55.177233 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.177239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.177410 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.177581 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.180441 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.180452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.180465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.180485 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.180555 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.180583 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.180786 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.180850 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.180942 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7069,"thread_start_us":6905,"threads_started":1}
I20260807 06:48:55.181438 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.158206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54906)
0807 06:48:55.176949 (+ 18743us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.176978 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:55.177096 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:55.178752 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:55.178757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:55.178786 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:55.180193 (+ 1407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:55.180196 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:55.180211 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:55.180259 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:55.180713 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:55.180717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:55.180722 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:48:55.180868 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 06:48:55.180923 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18576,"thread_start_us":15522,"threads_started":1}
I20260807 06:48:55.185055 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 450f37ff8b934cae8803d231ea9d45d5 (DEFAULT_TABLE table=test-table [id=a38118fb077a488dbc8f956f78a961a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:55.185235 2615 tablet_service.cc:1467] Processing CreateTablet for tablet 450f37ff8b934cae8803d231ea9d45d5 (DEFAULT_TABLE table=test-table [id=a38118fb077a488dbc8f956f78a961a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:55.185299 2434 tablet_service.cc:1467] Processing CreateTablet for tablet 450f37ff8b934cae8803d231ea9d45d5 (DEFAULT_TABLE table=test-table [id=a38118fb077a488dbc8f956f78a961a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:48:55.186712 2434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450f37ff8b934cae8803d231ea9d45d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:55.186730 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450f37ff8b934cae8803d231ea9d45d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:55.186735 2615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450f37ff8b934cae8803d231ea9d45d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:55.200178 2903 tablet_bootstrap.cc:492] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: Bootstrap starting.
I20260807 06:48:55.202570 2908 tablet_bootstrap.cc:492] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Bootstrap starting.
I20260807 06:48:55.204124 2904 tablet_bootstrap.cc:492] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Bootstrap starting.
I20260807 06:48:55.206795 2903 tablet_bootstrap.cc:654] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:55.206869 2908 tablet_bootstrap.cc:654] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:55.207976 2908 log.cc:824] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:55.208097 2904 tablet_bootstrap.cc:654] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:55.208249 2903 log.cc:824] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:55.209597 2904 log.cc:824] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:55.211313 2908 tablet_bootstrap.cc:492] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: No bootstrap required, opened a new log
I20260807 06:48:55.211673 2908 ts_tablet_manager.cc:1397] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 06:48:55.212522 2903 tablet_bootstrap.cc:492] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: No bootstrap required, opened a new log
I20260807 06:48:55.212842 2903 ts_tablet_manager.cc:1397] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260807 06:48:55.214573 2904 tablet_bootstrap.cc:492] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: No bootstrap required, opened a new log
I20260807 06:48:55.214922 2904 ts_tablet_manager.cc:1397] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260807 06:48:55.220258 2908 raft_consensus.cc:348] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.220790 2908 raft_consensus.cc:374] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:55.220904 2908 raft_consensus.cc:729] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36374125e1d448cd821c1f14c7dab9b7, State: Initialized, Role: FOLLOWER
I20260807 06:48:55.221601 2908 consensus_queue.cc:260] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.222909 2711 heartbeater.cc:503] Master 127.27.54.126:34547 was elected leader, sending a full tablet report...
I20260807 06:48:55.223483 2908 ts_tablet_manager.cc:1428] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260807 06:48:55.225654 2903 raft_consensus.cc:348] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.226142 2903 raft_consensus.cc:374] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:55.226236 2903 raft_consensus.cc:729] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2121892450e4941ac8edf62dbe950d7, State: Initialized, Role: FOLLOWER
I20260807 06:48:55.226704 2904 raft_consensus.cc:348] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.227217 2904 raft_consensus.cc:374] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:55.227061 2903 consensus_queue.cc:260] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.227345 2904 raft_consensus.cc:729] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3350edfa1640463d951abb1d412d70d4, State: Initialized, Role: FOLLOWER
I20260807 06:48:55.228190 2904 consensus_queue.cc:260] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.229451 2903 ts_tablet_manager.cc:1428] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260807 06:48:55.230441 2879 heartbeater.cc:503] Master 127.27.54.126:34547 was elected leader, sending a full tablet report...
I20260807 06:48:55.232264 2904 ts_tablet_manager.cc:1428] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260807 06:48:55.263324 2917 raft_consensus.cc:482] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:48:55.263760 2917 raft_consensus.cc:504] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.265638 2917 leader_election.cc:288] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3350edfa1640463d951abb1d412d70d4 (127.27.54.67:43229), 36374125e1d448cd821c1f14c7dab9b7 (127.27.54.66:41277)
I20260807 06:48:55.269569 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.266117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:47657 (local address 127.27.54.66:41277)
0807 06:48:55.266319 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.266326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.267363 (+ 1037us) server_negotiation.cc:407] Connection header received
0807 06:48:55.267581 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.267587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.267650 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.267758 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.268880 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.268886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.268891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.268912 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.268952 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.268964 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.269095 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.269170 (+ 75us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.269243 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:48:55.269665 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.266141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:54285 (local address 127.27.54.67:43229)
0807 06:48:55.266361 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.266368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.267320 (+ 952us) server_negotiation.cc:407] Connection header received
0807 06:48:55.267518 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.267524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.267586 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.267704 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.268972 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.268977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.268982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.269012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.269050 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.269065 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.269207 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.269307 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.269374 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 06:48:55.269884 2921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.266117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:43229 (local address 127.27.54.65:54285)
0807 06:48:55.267191 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.267211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:55.267371 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:55.267728 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:55.267733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:55.267759 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:55.268768 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:55.268770 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:55.268783 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:55.268820 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:55.269177 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:55.269184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:55.269190 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:48:55.269346 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 06:48:55.269397 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":579,"threads_started":1}
I20260807 06:48:55.269946 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.266206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:41277 (local address 127.27.54.65:47657)
0807 06:48:55.267219 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.267238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:55.267435 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:55.267785 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:55.267790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:55.267813 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:48:55.268660 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:48:55.268662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:48:55.268677 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:48:55.268722 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:48:55.269062 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:48:55.269067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:48:55.269072 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 06:48:55.269220 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 06:48:55.269275 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"spinlock_wait_cycles":22400,"thread_start_us":600,"threads_started":1}
I20260807 06:48:55.270983 2823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450f37ff8b934cae8803d231ea9d45d5" candidate_uuid: "d2121892450e4941ac8edf62dbe950d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3350edfa1640463d951abb1d412d70d4" is_pre_election: true
I20260807 06:48:55.271059 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450f37ff8b934cae8803d231ea9d45d5" candidate_uuid: "d2121892450e4941ac8edf62dbe950d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36374125e1d448cd821c1f14c7dab9b7" is_pre_election: true
I20260807 06:48:55.271528 2823 raft_consensus.cc:2454] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2121892450e4941ac8edf62dbe950d7 in term 0.
I20260807 06:48:55.271577 2647 raft_consensus.cc:2454] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2121892450e4941ac8edf62dbe950d7 in term 0.
I20260807 06:48:55.272253 2366 leader_election.cc:302] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3350edfa1640463d951abb1d412d70d4, 36374125e1d448cd821c1f14c7dab9b7, d2121892450e4941ac8edf62dbe950d7; no voters:
I20260807 06:48:55.272658 2917 raft_consensus.cc:2781] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:48:55.272828 2917 raft_consensus.cc:482] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:48:55.272951 2917 raft_consensus.cc:3037] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:55.275337 2917 raft_consensus.cc:504] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.275983 2917 leader_election.cc:288] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3350edfa1640463d951abb1d412d70d4 (127.27.54.67:43229), 36374125e1d448cd821c1f14c7dab9b7 (127.27.54.66:41277)
I20260807 06:48:55.276569 2647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450f37ff8b934cae8803d231ea9d45d5" candidate_uuid: "d2121892450e4941ac8edf62dbe950d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36374125e1d448cd821c1f14c7dab9b7"
I20260807 06:48:55.276571 2823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450f37ff8b934cae8803d231ea9d45d5" candidate_uuid: "d2121892450e4941ac8edf62dbe950d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3350edfa1640463d951abb1d412d70d4"
I20260807 06:48:55.276793 2647 raft_consensus.cc:3037] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:55.276809 2823 raft_consensus.cc:3037] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:55.279232 2647 raft_consensus.cc:2454] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2121892450e4941ac8edf62dbe950d7 in term 1.
I20260807 06:48:55.279237 2823 raft_consensus.cc:2454] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2121892450e4941ac8edf62dbe950d7 in term 1.
I20260807 06:48:55.279775 2369 leader_election.cc:302] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36374125e1d448cd821c1f14c7dab9b7, d2121892450e4941ac8edf62dbe950d7; no voters:
I20260807 06:48:55.280184 2917 raft_consensus.cc:2781] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:55.280714 2917 raft_consensus.cc:686] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 1 LEADER]: Becoming Leader. State: Replica: d2121892450e4941ac8edf62dbe950d7, State: Running, Role: LEADER
I20260807 06:48:55.281216 2917 consensus_queue.cc:237] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } }
I20260807 06:48:55.286396 2251 catalog_manager.cc:5495] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2121892450e4941ac8edf62dbe950d7 (127.27.54.65). New cstate: current_term: 1 leader_uuid: "d2121892450e4941ac8edf62dbe950d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2121892450e4941ac8edf62dbe950d7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34527 } health_report { overall_health: HEALTHY } } }
I20260807 06:48:55.328297 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.325947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.27.54.65:34527)
0807 06:48:55.326153 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.326159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.326181 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:48:55.326361 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.326364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.326453 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.326574 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.327635 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.327639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.327643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.327658 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.327691 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.327703 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.327843 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.327959 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.328021 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:48:55.336813 2823 raft_consensus.cc:1261] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 1 FOLLOWER]: Refusing update from remote peer d2121892450e4941ac8edf62dbe950d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:55.336804 2647 raft_consensus.cc:1261] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 1 FOLLOWER]: Refusing update from remote peer d2121892450e4941ac8edf62dbe950d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:48:55.337878 2923 consensus_queue.cc:1035] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3350edfa1640463d951abb1d412d70d4" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:55.338229 2917 consensus_queue.cc:1035] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36374125e1d448cd821c1f14c7dab9b7" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:48:55.349781 2926 mvcc.cc:205] Tried to move back new op lower bound from 7315805533529174016 to 7315805533315665920. Current Snapshot: MvccSnapshot[applied={T|T < 7315805533529174016}]
I20260807 06:48:55.365628 2251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:55.365928 2251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:48:55.370092 2251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 269fe4638cf340478008780c449126e6: Sending DeleteTablet for 3 replicas of tablet 450f37ff8b934cae8803d231ea9d45d5
I20260807 06:48:55.371793 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 450f37ff8b934cae8803d231ea9d45d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:55 UTC) from {username='slave'} at 127.0.0.1:49502
I20260807 06:48:55.371829 2434 tablet_service.cc:1514] Processing DeleteTablet for tablet 450f37ff8b934cae8803d231ea9d45d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:55 UTC) from {username='slave'} at 127.0.0.1:54906
I20260807 06:48:55.371784 2615 tablet_service.cc:1514] Processing DeleteTablet for tablet 450f37ff8b934cae8803d231ea9d45d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:48:55 UTC) from {username='slave'} at 127.0.0.1:49394
I20260807 06:48:55.375731 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 2343
I20260807 06:48:55.384539 2939 tablet_replica.cc:330] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: stopping tablet replica
I20260807 06:48:55.385202 2939 raft_consensus.cc:2229] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:48:55.386232 2939 raft_consensus.cc:2258] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:55.387382 2941 tablet_replica.cc:330] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: stopping tablet replica
I20260807 06:48:55.387871 2942 tablet_replica.cc:330] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: stopping tablet replica
I20260807 06:48:55.387990 2941 raft_consensus.cc:2229] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:55.388510 2942 raft_consensus.cc:2229] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:48:55.388881 2941 raft_consensus.cc:2258] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:55.389138 2942 raft_consensus.cc:2258] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:48:55.391525 2942 ts_tablet_manager.cc:1905] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:55.392791 2941 ts_tablet_manager.cc:1905] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:55.394021 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.377782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.27.54.65:34527)
0807 06:48:55.378046 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.378056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.390998 (+ 12942us) server_negotiation.cc:407] Connection header received
0807 06:48:55.391290 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.391296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.391375 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.391496 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.393013 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.393021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.393026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.393051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.393099 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.393118 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.393266 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.393587 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.393667 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 06:48:55.395313 2513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:55.391533 2939 ts_tablet_manager.cc:1905] T 450f37ff8b934cae8803d231ea9d45d5 P d2121892450e4941ac8edf62dbe950d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:48:55.400642 2942 ts_tablet_manager.cc:1918] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:55.400827 2942 log.cc:1196] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/450f37ff8b934cae8803d231ea9d45d5
I20260807 06:48:55.401387 2942 ts_tablet_manager.cc:1939] T 450f37ff8b934cae8803d231ea9d45d5 P 3350edfa1640463d951abb1d412d70d4: Deleting consensus metadata
I20260807 06:48:55.402325 2941 ts_tablet_manager.cc:1918] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:48:55.402529 2941 log.cc:1196] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/450f37ff8b934cae8803d231ea9d45d5
I20260807 06:48:55.403124 2941 ts_tablet_manager.cc:1939] T 450f37ff8b934cae8803d231ea9d45d5 P 36374125e1d448cd821c1f14c7dab9b7: Deleting consensus metadata
I20260807 06:48:55.403540 2228 catalog_manager.cc:4841] TS 3350edfa1640463d951abb1d412d70d4 (127.27.54.67:43229): tablet 450f37ff8b934cae8803d231ea9d45d5 (table test-table [id=a38118fb077a488dbc8f956f78a961a2]) successfully deleted
I20260807 06:48:55.404860 2232 catalog_manager.cc:4841] TS 36374125e1d448cd821c1f14c7dab9b7 (127.27.54.66:41277): tablet 450f37ff8b934cae8803d231ea9d45d5 (table test-table [id=a38118fb077a488dbc8f956f78a961a2]) successfully deleted
I20260807 06:48:55.494475 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 2343
I20260807 06:48:55.522282 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 2522
W20260807 06:48:55.522408 2231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.54.65:34527 (error 108)
I20260807 06:48:55.524806 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.524171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54938)
0807 06:48:55.524390 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.524617 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 06:48:55.525104 2231 catalog_manager.cc:4584] TS d2121892450e4941ac8edf62dbe950d7 (127.27.54.65:34527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 450f37ff8b934cae8803d231ea9d45d5: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
I20260807 06:48:55.525842 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.523262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.27.54.66:41277)
0807 06:48:55.523669 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.523674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.523693 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:48:55.523854 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.523858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.523995 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.524112 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.525179 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.525185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.525188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.525205 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.525238 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.525250 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.525392 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.525493 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.525560 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260807 06:48:55.526556 2705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:55.579068 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.578418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54948)
0807 06:48:55.578728 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.578859 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260807 06:48:55.580276 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.579867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54964)
0807 06:48:55.580035 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.580137 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:48:55.617547 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 2522
I20260807 06:48:55.641178 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.640655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54966)
0807 06:48:55.640898 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.641004 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260807 06:48:55.642354 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.641981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54974)
0807 06:48:55.642145 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.642221 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 06:48:55.644091 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 2715
I20260807 06:48:55.647140 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.644868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.27.54.67:43229)
0807 06:48:55.645073 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:55.645079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:55.645100 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:48:55.645269 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:55.645278 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:55.645367 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:55.645481 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:55.646382 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:55.646388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:55.646392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:55.646418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:55.646450 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:55.646462 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:55.646599 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:55.646770 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 06:48:55.646846 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:48:55.647804 2871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:55.723182 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.722576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54978)
0807 06:48:55.722850 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.722984 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260807 06:48:55.724470 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:55.723972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34527 (local address 127.0.0.1:54984)
0807 06:48:55.724228 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:55.724319 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260807 06:48:55.742559 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 2715
I20260807 06:48:55.766773 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 2201
I20260807 06:48:55.768014 2301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:55.880508 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 2201
2026-08-07T06:48:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2864 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 06:48:55.934481 27865 mini_oidc.cc:152] Starting JWKS server
I20260807 06:48:55.938575 27865 webserver.cc:460] Webserver started at https://127.0.0.1:33517/ using document root <none> and password file <none>
I20260807 06:48:55.939067 27865 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 06:48:55.939997 27865 webserver.cc:460] Webserver started at http://127.0.0.1:39431/ using document root <none> and password file <none>
2026-08-07T06:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:55Z Disabled control of system clock
I20260807 06:48:55.981778 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:45261
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:46169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:45261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33517/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/testchainca.cert with env {}
W20260807 06:48:56.347420 2961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:56.347780 2961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:56.347877 2961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:56.355602 2961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:56.355748 2961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:56.355790 2961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:56.355852 2961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:56.355898 2961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:48:56.355963 2961 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:48:56.356006 2961 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33517/jwks.json
I20260807 06:48:56.366724 2961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:45261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33517/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:56.368110 2961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:56.369527 2961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:56.382298 2971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:56.382341 2975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:56.382472 2970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:56.383116 2961 server_base.cc:1034] running on GCE node
I20260807 06:48:56.384059 2961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:56.385591 2961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:56.386894 2961 hybrid_clock.cc:648] HybridClock initialized: now 1786085336386843 us; error 34 us; skew 500 ppm
I20260807 06:48:56.387301 2961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:56.392899 2961 webserver.cc:460] Webserver started at http://127.27.54.126:38493/ using document root <none> and password file <none>
I20260807 06:48:56.393750 2961 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:56.393900 2961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:56.394284 2961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:56.397775 2961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "986279e4b6c04b10af7c8b4b6546fe41"
format_stamp: "Formatted at 2026-08-07 06:48:56 on dist-test-slave-3kk6"
server_key: "29710141e9aef51b9b51eb3a6ce31355"
server_key_iv: "ff4c9aff1a93daa4c5c8513c8211b6b6"
server_key_version: "encryptionkey@0"
I20260807 06:48:56.398729 2961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "986279e4b6c04b10af7c8b4b6546fe41"
format_stamp: "Formatted at 2026-08-07 06:48:56 on dist-test-slave-3kk6"
server_key: "29710141e9aef51b9b51eb3a6ce31355"
server_key_iv: "ff4c9aff1a93daa4c5c8513c8211b6b6"
server_key_version: "encryptionkey@0"
I20260807 06:48:56.404222 2961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 06:48:56.407936 2982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:56.409428 2961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:56.409649 2961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "986279e4b6c04b10af7c8b4b6546fe41"
format_stamp: "Formatted at 2026-08-07 06:48:56 on dist-test-slave-3kk6"
server_key: "29710141e9aef51b9b51eb3a6ce31355"
server_key_iv: "ff4c9aff1a93daa4c5c8513c8211b6b6"
server_key_version: "encryptionkey@0"
I20260807 06:48:56.409909 2961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:56.426081 2961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:56.427083 2961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:56.427436 2961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:56.448362 2961 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:45261
I20260807 06:48:56.448396 3062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:45261 every 8 connection(s)
I20260807 06:48:56.449951 2961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:56.450909 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 2961
I20260807 06:48:56.451140 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:56.451859 27865 external_mini_cluster.cc:1428] Setting key 035b2b6bc384df31b17bc11046c9397f
I20260807 06:48:56.454917 3065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:56.469467 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:56.453968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.27.54.126:45261)
0807 06:48:56.455246 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:56.455262 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:56.455320 (+ 58us) server_negotiation.cc:407] Connection header received
0807 06:48:56.457819 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:56.457852 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:56.458763 (+ 911us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:56.459293 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:56.459910 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:56.462036 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:56.463691 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:56.464776 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:56.464989 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:56.466285 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:56.466301 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:56.466322 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:56.467215 (+ 893us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:56.467320 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:56.467349 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:56.467494 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:56.467710 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 06:48:56.468102 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":435,"threads_started":1}
I20260807 06:48:56.470373 3065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41: Bootstrap starting.
I20260807 06:48:56.473762 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:48:56.474329 3065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:56.475440 3065 log.cc:824] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:56.478833 3065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41: No bootstrap required, opened a new log
I20260807 06:48:56.486922 3065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986279e4b6c04b10af7c8b4b6546fe41" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45261 } }
I20260807 06:48:56.487520 3065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:56.487650 3065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986279e4b6c04b10af7c8b4b6546fe41, State: Initialized, Role: FOLLOWER
I20260807 06:48:56.488258 3065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986279e4b6c04b10af7c8b4b6546fe41" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45261 } }
I20260807 06:48:56.488488 3065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:56.488592 3065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:56.488759 3065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:56.491163 3065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986279e4b6c04b10af7c8b4b6546fe41" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45261 } }
I20260807 06:48:56.491683 3065 leader_election.cc:302] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 986279e4b6c04b10af7c8b4b6546fe41; no voters:
I20260807 06:48:56.492254 3065 leader_election.cc:288] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:56.492478 3071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:56.493281 3071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [term 1 LEADER]: Becoming Leader. State: Replica: 986279e4b6c04b10af7c8b4b6546fe41, State: Running, Role: LEADER
I20260807 06:48:56.493819 3071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986279e4b6c04b10af7c8b4b6546fe41" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45261 } }
I20260807 06:48:56.494580 3065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:56.497073 3075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 986279e4b6c04b10af7c8b4b6546fe41. Latest consensus state: current_term: 1 leader_uuid: "986279e4b6c04b10af7c8b4b6546fe41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986279e4b6c04b10af7c8b4b6546fe41" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45261 } } }
I20260807 06:48:56.497943 3075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:56.497519 3072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "986279e4b6c04b10af7c8b4b6546fe41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "986279e4b6c04b10af7c8b4b6546fe41" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45261 } } }
I20260807 06:48:56.498144 3072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:56.499835 3086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:56.505038 3086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:56.517190 3086 catalog_manager.cc:1347] Generated new cluster ID: 367715fc77f748049acde20cd0a6ca61
I20260807 06:48:56.517342 3086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:56.527634 3086 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:56.529035 3086 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:56.538921 3086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 986279e4b6c04b10af7c8b4b6546fe41: Generated new TSK 0
I20260807 06:48:56.539943 3086 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 06:48:56.586918 3111 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 06:48:56.587401 3111 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 06:48:56.588739 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:56.564551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.27.54.126:45261)
0807 06:48:56.564779 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:56.564787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:56.565397 (+ 610us) server_negotiation.cc:407] Connection header received
0807 06:48:56.565802 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:56.565808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:56.565918 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:56.566104 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:48:56.566704 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:56.567969 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:56.569517 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:56.570257 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:56.570401 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:56.571107 (+ 706us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:48:56.587866 (+ 16759us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 06:48:56.588387 (+ 521us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 06:48:56.588907 3068 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 06:48:56.591318 3110 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:48:56.564410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:45261 (local address 127.0.0.1:56796)
0807 06:48:56.565262 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:56.565285 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:56.565470 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:56.566147 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:56.566151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:56.566162 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:48:56.566515 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:56.566524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:56.568153 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:56.568157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:56.569355 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:56.569367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:56.570470 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:56.570475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:56.570615 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:56.570759 (+ 144us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:48:56.588316 (+ 17557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 06:48:56.588683 (+ 367us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.54.126:45261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":461,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.54.126:45261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 06:48:56.594548 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 2961
I20260807 06:48:56.596319 3056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:56.722714 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 2961
2026-08-07T06:48:56Z chronyd exiting
I20260807 06:48:56.780124 27865 test_util.cc:175] -----------------------------------------------
I20260807 06:48:56.780263 27865 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786085287136147-27865-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (852 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 06:48:56.784443 27865 mini_oidc.cc:152] Starting JWKS server
I20260807 06:48:56.786489 27865 webserver.cc:460] Webserver started at https://127.0.0.1:36949/ using document root <none> and password file <none>
I20260807 06:48:56.786834 27865 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 06:48:56.787529 27865 webserver.cc:460] Webserver started at http://127.0.0.1:39995/ using document root <none> and password file <none>
2026-08-07T06:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:56Z Disabled control of system clock
I20260807 06:48:56.827766 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:37921
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:37921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36949/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/testchainca.cert with env {}
W20260807 06:48:57.189073 3122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:57.189469 3122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:57.189630 3122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:57.197278 3122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:57.197470 3122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:57.197568 3122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:57.197690 3122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:57.197786 3122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:48:57.197889 3122 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:48:57.197968 3122 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36949/jwks.json
I20260807 06:48:57.209347 3122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:37921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36949/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:57.210986 3122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:57.212579 3122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:57.226130 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:57.226171 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:57.226094 3137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:57.227555 3122 server_base.cc:1034] running on GCE node
I20260807 06:48:57.228538 3122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:57.230154 3122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:57.231379 3122 hybrid_clock.cc:648] HybridClock initialized: now 1786085337231327 us; error 51 us; skew 500 ppm
I20260807 06:48:57.231819 3122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:57.234438 3122 webserver.cc:460] Webserver started at http://127.27.54.126:43503/ using document root <none> and password file <none>
I20260807 06:48:57.235286 3122 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:57.235416 3122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:57.235800 3122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:57.239099 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "cf843164542248b7a925d9efaf80e3c8"
format_stamp: "Formatted at 2026-08-07 06:48:57 on dist-test-slave-3kk6"
server_key: "ffaf90690b3640f25dc1d2348acb943b"
server_key_iv: "c526dd4b9b154127f0e2a357a555a0d7"
server_key_version: "encryptionkey@0"
I20260807 06:48:57.240010 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "cf843164542248b7a925d9efaf80e3c8"
format_stamp: "Formatted at 2026-08-07 06:48:57 on dist-test-slave-3kk6"
server_key: "ffaf90690b3640f25dc1d2348acb943b"
server_key_iv: "c526dd4b9b154127f0e2a357a555a0d7"
server_key_version: "encryptionkey@0"
I20260807 06:48:57.245659 3122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:48:57.249604 3145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:57.251267 3122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:48:57.251473 3122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "cf843164542248b7a925d9efaf80e3c8"
format_stamp: "Formatted at 2026-08-07 06:48:57 on dist-test-slave-3kk6"
server_key: "ffaf90690b3640f25dc1d2348acb943b"
server_key_iv: "c526dd4b9b154127f0e2a357a555a0d7"
server_key_version: "encryptionkey@0"
I20260807 06:48:57.251741 3122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:57.275511 3122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:57.276495 3122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:57.276861 3122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:57.297202 3122 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:37921
I20260807 06:48:57.297220 3220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:37921 every 8 connection(s)
I20260807 06:48:57.298784 3122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:57.302719 3221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:57.305912 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 3122
I20260807 06:48:57.306146 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:57.306937 27865 external_mini_cluster.cc:1428] Setting key d585ba43211c6ad877ebf81ea0e1be11
I20260807 06:48:57.316536 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8: Bootstrap starting.
I20260807 06:48:57.320673 3221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:57.321962 3221 log.cc:824] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:57.322184 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:57.309200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.27.54.126:37921)
0807 06:48:57.310360 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:57.310372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:57.310419 (+ 47us) server_negotiation.cc:407] Connection header received
0807 06:48:57.312212 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:57.312238 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:57.312788 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:57.313184 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:57.313914 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:57.315329 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:57.317196 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:57.317923 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:57.318067 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:57.319556 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:57.319564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:57.319580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:57.320196 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:57.320304 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:57.320327 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:57.320427 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:57.320720 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 06:48:57.321095 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":450,"threads_started":1}
I20260807 06:48:57.325522 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8: No bootstrap required, opened a new log
I20260807 06:48:57.326691 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:48:57.334568 3221 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf843164542248b7a925d9efaf80e3c8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37921 } }
I20260807 06:48:57.335153 3221 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:57.335307 3221 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf843164542248b7a925d9efaf80e3c8, State: Initialized, Role: FOLLOWER
I20260807 06:48:57.336045 3221 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf843164542248b7a925d9efaf80e3c8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37921 } }
I20260807 06:48:57.336270 3221 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:57.336382 3221 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:57.336509 3221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:57.338866 3221 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf843164542248b7a925d9efaf80e3c8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37921 } }
I20260807 06:48:57.339403 3221 leader_election.cc:302] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf843164542248b7a925d9efaf80e3c8; no voters:
I20260807 06:48:57.339960 3221 leader_election.cc:288] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:57.340142 3230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:57.340919 3230 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [term 1 LEADER]: Becoming Leader. State: Replica: cf843164542248b7a925d9efaf80e3c8, State: Running, Role: LEADER
I20260807 06:48:57.341497 3230 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf843164542248b7a925d9efaf80e3c8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37921 } }
I20260807 06:48:57.342216 3221 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:57.344264 3234 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf843164542248b7a925d9efaf80e3c8. Latest consensus state: current_term: 1 leader_uuid: "cf843164542248b7a925d9efaf80e3c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf843164542248b7a925d9efaf80e3c8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37921 } } }
I20260807 06:48:57.344730 3234 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:57.344545 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf843164542248b7a925d9efaf80e3c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf843164542248b7a925d9efaf80e3c8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37921 } } }
I20260807 06:48:57.344990 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:57.346045 3237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:57.353665 3237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:57.364041 3237 catalog_manager.cc:1347] Generated new cluster ID: 13c6686aaa8447f3866893422706af99
I20260807 06:48:57.364192 3237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:57.384341 3237 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:57.385774 3237 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:57.404870 3237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf843164542248b7a925d9efaf80e3c8: Generated new TSK 0
I20260807 06:48:57.405874 3237 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 06:48:57.439260 3264 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 06:48:57.439435 3264 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 06:48:57.440779 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:57.419031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.27.54.126:37921)
0807 06:48:57.419325 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:57.419335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:57.420018 (+ 683us) server_negotiation.cc:407] Connection header received
0807 06:48:57.420262 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:57.420268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:57.420370 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:57.420565 (+ 195us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:48:57.421082 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:57.422297 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:57.423891 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:57.424621 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:57.424763 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:57.425268 (+ 505us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:48:57.440014 (+ 14746us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 06:48:57.440454 (+ 440us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":110}
W20260807 06:48:57.440939 3227 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 06:48:57.441035 3263 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:48:57.418712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:37921 (local address 127.0.0.1:49226)
0807 06:48:57.419878 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:57.419896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:57.420051 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:57.420583 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:57.420587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:57.420596 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:48:57.420902 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:57.420911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:57.422508 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:57.422512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:57.423722 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:57.423735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:57.424803 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:57.424807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:57.424943 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:57.424979 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:48:57.440327 (+ 15348us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 06:48:57.440578 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.54.126:37921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":843,"threads_started":1}
I20260807 06:48:57.442754 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 3122
I20260807 06:48:57.444262 3215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:57.542799 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 3122
2026-08-07T06:48:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (825 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 06:48:57.610414 27865 mini_oidc.cc:152] Starting JWKS server
I20260807 06:48:57.612684 27865 webserver.cc:460] Webserver started at https://127.0.0.1:35481/ using document root <none> and password file <none>
I20260807 06:48:57.612977 27865 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 06:48:57.613754 27865 webserver.cc:460] Webserver started at http://127.0.0.1:34797/ using document root <none> and password file <none>
2026-08-07T06:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:57Z Disabled control of system clock
I20260807 06:48:57.655915 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:43731
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:43731
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35481/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 06:48:58.012121 3275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:58.012529 3275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:58.012697 3275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:58.020215 3275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:58.020395 3275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:58.020496 3275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:58.020620 3275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:58.020713 3275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:48:58.020807 3275 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:48:58.020888 3275 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35481/jwks.json
I20260807 06:48:58.031108 3275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:43731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35481/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:58.032618 3275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:58.034533 3275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:58.046800 3287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:58.046970 3289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:58.047417 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:58.048079 3275 server_base.cc:1034] running on GCE node
I20260807 06:48:58.048913 3275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:58.050442 3275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:58.051699 3275 hybrid_clock.cc:648] HybridClock initialized: now 1786085338051644 us; error 63 us; skew 500 ppm
I20260807 06:48:58.052065 3275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:58.054433 3275 webserver.cc:460] Webserver started at http://127.27.54.126:36669/ using document root <none> and password file <none>
I20260807 06:48:58.055226 3275 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:58.055368 3275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:58.055727 3275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:58.058985 3275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "0d5ec45539424e609e75e5e78a4f4d46"
format_stamp: "Formatted at 2026-08-07 06:48:58 on dist-test-slave-3kk6"
server_key: "8231098e6d2181ee59689bb3e3c61c8a"
server_key_iv: "0d8370301415484a6e4d62ff4079c5f1"
server_key_version: "encryptionkey@0"
I20260807 06:48:58.059878 3275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "0d5ec45539424e609e75e5e78a4f4d46"
format_stamp: "Formatted at 2026-08-07 06:48:58 on dist-test-slave-3kk6"
server_key: "8231098e6d2181ee59689bb3e3c61c8a"
server_key_iv: "0d8370301415484a6e4d62ff4079c5f1"
server_key_version: "encryptionkey@0"
I20260807 06:48:58.065590 3275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 06:48:58.069303 3304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:58.070842 3275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:48:58.071045 3275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "0d5ec45539424e609e75e5e78a4f4d46"
format_stamp: "Formatted at 2026-08-07 06:48:58 on dist-test-slave-3kk6"
server_key: "8231098e6d2181ee59689bb3e3c61c8a"
server_key_iv: "0d8370301415484a6e4d62ff4079c5f1"
server_key_version: "encryptionkey@0"
I20260807 06:48:58.071310 3275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:58.094249 3275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:58.095589 3275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:58.096051 3275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:58.119052 3275 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:43731
I20260807 06:48:58.119071 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:43731 every 8 connection(s)
I20260807 06:48:58.120518 3275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:58.124569 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:58.125623 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 3275
I20260807 06:48:58.125847 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:58.126770 27865 external_mini_cluster.cc:1428] Setting key a81b23a4470babc47342b199c9ec36a0
I20260807 06:48:58.139829 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46: Bootstrap starting.
I20260807 06:48:58.141455 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:58.128926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.27.54.126:43731)
0807 06:48:58.130101 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:58.130112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:58.130161 (+ 49us) server_negotiation.cc:407] Connection header received
0807 06:48:58.131896 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:58.131926 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:58.132465 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:58.132830 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:58.133462 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.134825 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.136820 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.137550 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.137687 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:58.139038 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:58.139046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:58.139061 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:58.139624 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:58.139703 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:58.139724 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:58.139817 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:58.140119 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 06:48:58.140479 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":419,"threads_started":1}
I20260807 06:48:58.143339 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:58.144553 3377 log.cc:824] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:58.145381 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:48:58.147519 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46: No bootstrap required, opened a new log
I20260807 06:48:58.155244 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ec45539424e609e75e5e78a4f4d46" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43731 } }
I20260807 06:48:58.155627 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:58.155709 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5ec45539424e609e75e5e78a4f4d46, State: Initialized, Role: FOLLOWER
I20260807 06:48:58.156301 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ec45539424e609e75e5e78a4f4d46" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43731 } }
I20260807 06:48:58.156515 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:58.156623 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:58.156762 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:58.159444 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ec45539424e609e75e5e78a4f4d46" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43731 } }
I20260807 06:48:58.159945 3377 leader_election.cc:302] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d5ec45539424e609e75e5e78a4f4d46; no voters:
I20260807 06:48:58.160696 3377 leader_election.cc:288] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:58.160789 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:58.161645 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [term 1 LEADER]: Becoming Leader. State: Replica: 0d5ec45539424e609e75e5e78a4f4d46, State: Running, Role: LEADER
I20260807 06:48:58.162320 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ec45539424e609e75e5e78a4f4d46" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43731 } }
I20260807 06:48:58.163147 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:58.165539 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d5ec45539424e609e75e5e78a4f4d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ec45539424e609e75e5e78a4f4d46" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43731 } } }
I20260807 06:48:58.165859 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5ec45539424e609e75e5e78a4f4d46. Latest consensus state: current_term: 1 leader_uuid: "0d5ec45539424e609e75e5e78a4f4d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ec45539424e609e75e5e78a4f4d46" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43731 } } }
I20260807 06:48:58.166128 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:58.166211 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:58.168577 3391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:58.174355 3391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:58.183663 3391 catalog_manager.cc:1347] Generated new cluster ID: ae9109799b6f4dc9b0d9ff45e279b11b
I20260807 06:48:58.183805 3391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:58.218596 3391 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:58.220654 3391 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:58.231740 3391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d5ec45539424e609e75e5e78a4f4d46: Generated new TSK 0
I20260807 06:48:58.232810 3391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:48:58.271265 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:58.263277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.27.54.126:43731)
0807 06:48:58.263475 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:58.263481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:58.263972 (+ 491us) server_negotiation.cc:407] Connection header received
0807 06:48:58.264214 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:58.264218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:58.264291 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:58.264451 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:48:58.264953 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.265683 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.267147 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.267620 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.267773 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:58.268295 (+ 522us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:48:58.270552 (+ 2257us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 06:48:58.270951 (+ 399us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 06:48:58.271605 3380 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44924: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 06:48:58.271824 3418 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:48:58.263092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:43731 (local address 127.0.0.1:44924)
0807 06:48:58.263862 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:58.263880 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:58.264010 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:58.264472 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:58.264477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:58.264490 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:48:58.264787 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:58.264796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.265829 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.265835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:58.266989 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:58.267001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.267827 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.267835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:58.267984 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:58.268020 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:48:58.270812 (+ 2792us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 06:48:58.271092 (+ 280us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.54.126:43731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35481/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":1,"thread_start_us":450,"threads_started":1}
I20260807 06:48:58.273005 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 3275
I20260807 06:48:58.274196 3371 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 06:48:58.283766 3419 webserver.cc:533] Webserver: sslize failed:
E20260807 06:48:58.283897 3419 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 06:48:58.370412 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 3275
2026-08-07T06:48:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (822 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T06:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:48:58Z Disabled control of system clock
I20260807 06:48:58.472919 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:45269
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:45269
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/testchainca.cert with env {}
W20260807 06:48:58.826354 3424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:48:58.826740 3424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:48:58.826928 3424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:48:58.834336 3424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:48:58.834481 3424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:48:58.834581 3424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:48:58.834712 3424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:48:58.834798 3424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 06:48:58.834896 3424 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 06:48:58.834980 3424 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 06:48:58.845177 3424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:45269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:48:58.846570 3424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:48:58.847958 3424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:48:58.860581 3435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:58.860581 3437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:48:58.860877 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:48:58.861755 3424 server_base.cc:1034] running on GCE node
I20260807 06:48:58.862591 3424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:48:58.864279 3424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:48:58.865486 3424 hybrid_clock.cc:648] HybridClock initialized: now 1786085338865447 us; error 31 us; skew 500 ppm
I20260807 06:48:58.865870 3424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:48:58.868249 3424 webserver.cc:460] Webserver started at http://127.27.54.126:44623/ using document root <none> and password file <none>
I20260807 06:48:58.868963 3424 fs_manager.cc:359] Metadata directory not provided
I20260807 06:48:58.869091 3424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:48:58.869467 3424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:48:58.872495 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "fa4466fd961f4fd2a2e5b5f2be10d164"
format_stamp: "Formatted at 2026-08-07 06:48:58 on dist-test-slave-3kk6"
server_key: "2704ca240e5ca5e56b75b06af986c4b7"
server_key_iv: "8a3212367c1d11dbd323c0d7487396ab"
server_key_version: "encryptionkey@0"
I20260807 06:48:58.873373 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "fa4466fd961f4fd2a2e5b5f2be10d164"
format_stamp: "Formatted at 2026-08-07 06:48:58 on dist-test-slave-3kk6"
server_key: "2704ca240e5ca5e56b75b06af986c4b7"
server_key_iv: "8a3212367c1d11dbd323c0d7487396ab"
server_key_version: "encryptionkey@0"
I20260807 06:48:58.878899 3424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:48:58.882602 3447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:48:58.884176 3424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:48:58.884414 3424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "fa4466fd961f4fd2a2e5b5f2be10d164"
format_stamp: "Formatted at 2026-08-07 06:48:58 on dist-test-slave-3kk6"
server_key: "2704ca240e5ca5e56b75b06af986c4b7"
server_key_iv: "8a3212367c1d11dbd323c0d7487396ab"
server_key_version: "encryptionkey@0"
I20260807 06:48:58.884690 3424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:48:58.902448 3424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:48:58.903425 3424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:48:58.903750 3424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:48:58.923311 3424 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:45269
I20260807 06:48:58.923331 3520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:45269 every 8 connection(s)
I20260807 06:48:58.924763 3424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:48:58.928668 3522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:48:58.931062 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 3424
I20260807 06:48:58.931319 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:48:58.931994 27865 external_mini_cluster.cc:1428] Setting key 0d2ee00e24768fcf415f9a40d3acee9d
I20260807 06:48:58.941082 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164: Bootstrap starting.
I20260807 06:48:58.945355 3522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164: Neither blocks nor log segments found. Creating new log.
I20260807 06:48:58.946631 3522 log.cc:824] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164: Log is configured to *not* fsync() on all Append() calls
I20260807 06:48:58.946982 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:58.934126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.27.54.126:45269)
0807 06:48:58.935253 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:58.935265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:58.935311 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:48:58.937036 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:58.937059 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:58.937581 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:58.937958 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:48:58.938642 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.940055 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.941873 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:58.942557 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:58.942789 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:58.944154 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:48:58.944180 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:48:58.944204 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:48:58.945057 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:48:58.945205 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:48:58.945235 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:48:58.945328 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 06:48:58.945626 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 06:48:58.945883 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":398,"threads_started":1}
I20260807 06:48:58.950850 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164: No bootstrap required, opened a new log
I20260807 06:48:58.951920 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:48:58.959745 3522 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45269 } }
I20260807 06:48:58.960145 3522 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:48:58.960223 3522 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4466fd961f4fd2a2e5b5f2be10d164, State: Initialized, Role: FOLLOWER
I20260807 06:48:58.960812 3522 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45269 } }
I20260807 06:48:58.961031 3522 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:48:58.961136 3522 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:48:58.961274 3522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:48:58.963598 3522 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45269 } }
I20260807 06:48:58.964218 3522 leader_election.cc:302] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4466fd961f4fd2a2e5b5f2be10d164; no voters:
I20260807 06:48:58.964867 3522 leader_election.cc:288] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:48:58.965080 3533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:48:58.965873 3533 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [term 1 LEADER]: Becoming Leader. State: Replica: fa4466fd961f4fd2a2e5b5f2be10d164, State: Running, Role: LEADER
I20260807 06:48:58.966496 3533 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45269 } }
I20260807 06:48:58.967293 3522 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:48:58.969655 3535 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45269 } } }
I20260807 06:48:58.970021 3535 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:58.969708 3534 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4466fd961f4fd2a2e5b5f2be10d164. Latest consensus state: current_term: 1 leader_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4466fd961f4fd2a2e5b5f2be10d164" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45269 } } }
I20260807 06:48:58.970139 3534 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164 [sys.catalog]: This master's current role is: LEADER
I20260807 06:48:58.973929 3541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:48:58.979400 3541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:48:58.989231 3541 catalog_manager.cc:1347] Generated new cluster ID: 86a461b28d6643c08c1567578f2bfcf7
I20260807 06:48:58.989408 3541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:48:59.002089 3541 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:48:59.003559 3541 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:48:59.012805 3541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4466fd961f4fd2a2e5b5f2be10d164: Generated new TSK 0
I20260807 06:48:59.013710 3541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:48:59.166241 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:48:59.027521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.27.54.126:45269)
0807 06:48:59.027809 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 06:48:59.027817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:48:59.028246 (+ 429us) server_negotiation.cc:407] Connection header received
0807 06:48:59.028512 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:48:59.028517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:48:59.028601 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:48:59.028788 (+ 187us) server_negotiation.cc:226] Negotiated authn=JWT
0807 06:48:59.029408 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:59.030170 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:59.032361 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:48:59.032809 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:48:59.032931 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:59.033558 (+ 627us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 06:48:59.165588 (+132030us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 06:48:59.165953 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":79}
W20260807 06:48:59.166440 3528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 06:48:59.166780 3562 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:48:59.027412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:45269 (local address 127.0.0.1:41802)
0807 06:48:59.028092 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0807 06:48:59.028115 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:48:59.028291 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:48:59.028846 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:48:59.028852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:48:59.028866 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 06:48:59.029210 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:59.029224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:59.030332 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:59.030338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:59.032197 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:48:59.032214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:48:59.033042 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:48:59.033050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:48:59.033235 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:48:59.033286 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 06:48:59.165926 (+132640us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 06:48:59.166293 (+ 367us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.54.126:45269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":354,"threads_started":1}
I20260807 06:48:59.168673 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 3424
I20260807 06:48:59.169874 3515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:48:59.276043 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 3424
2026-08-07T06:48:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (884 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (454 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1595 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:49:01 dist-test-slave-3kk6 krb5kdc[3588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:49:01 dist-test-slave-3kk6 krb5kdc[3588](info): set up 2 sockets
Aug 07 06:49:01 dist-test-slave-3kk6 krb5kdc[3588](info): commencing operation
krb5kdc: starting...
W20260807 06:49:02.633164 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:49:02 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:49:02.920660 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38833
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:03.269601 3604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:03.269919 3604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:03.270099 3604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:03.277489 3604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:03.277608 3604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:03.277740 3604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:03.277848 3604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:03.277928 3604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:03.287966 3604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:38833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:03.289390 3604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:03.290833 3604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:03.303413 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:03.303493 3615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:03.303496 3620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:03.304776 3604 server_base.cc:1034] running on GCE node
I20260807 06:49:03.305644 3604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:03.307232 3604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:03.308444 3604 hybrid_clock.cc:648] HybridClock initialized: now 1786085343308394 us; error 38 us; skew 500 ppm
I20260807 06:49:03.308805 3604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:03 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085343, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:03.312105 3604 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:49:03.313736 3604 webserver.cc:460] Webserver started at http://127.27.54.126:38963/ using document root <none> and password file <none>
I20260807 06:49:03.314463 3604 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:03.314618 3604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:03.314990 3604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:03.318470 3604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e"
format_stamp: "Formatted at 2026-08-07 06:49:03 on dist-test-slave-3kk6"
server_key: "055b5bd9d0450c7c53b7d034aaf7e8b9"
server_key_iv: "05a7636768ab31c968bf01e25998b4c5"
server_key_version: "encryptionkey@0"
I20260807 06:49:03.319478 3604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e"
format_stamp: "Formatted at 2026-08-07 06:49:03 on dist-test-slave-3kk6"
server_key: "055b5bd9d0450c7c53b7d034aaf7e8b9"
server_key_iv: "05a7636768ab31c968bf01e25998b4c5"
server_key_version: "encryptionkey@0"
I20260807 06:49:03.324887 3604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:49:03.328536 3634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:03.329922 3604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:49:03.330116 3604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e"
format_stamp: "Formatted at 2026-08-07 06:49:03 on dist-test-slave-3kk6"
server_key: "055b5bd9d0450c7c53b7d034aaf7e8b9"
server_key_iv: "05a7636768ab31c968bf01e25998b4c5"
server_key_version: "encryptionkey@0"
I20260807 06:49:03.330334 3604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:03.349113 3604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:03.353415 3604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:03.353741 3604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:03.373463 3604 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:38833
I20260807 06:49:03.373504 3703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:38833 every 8 connection(s)
I20260807 06:49:03.375032 3604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:03.378795 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 3604
I20260807 06:49:03.379078 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:03.379518 3704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:03.379973 27865 external_mini_cluster.cc:1428] Setting key 2f7171f3fa6f2656799dfa1e80ddc293
I20260807 06:49:03.395090 3704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e: Bootstrap starting.
Aug 07 06:49:03 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:03.399277 3704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:03.400695 3704 log.cc:824] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:03.404439 3704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e: No bootstrap required, opened a new log
I20260807 06:49:03.405908 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:03.382118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.27.54.126:38833)
0807 06:49:03.383068 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:03.383083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:03.383126 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:49:03.384914 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:03.384939 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:03.385370 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:03.385732 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:03.387639 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:03.388945 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:03.390806 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:03.391538 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:03.391702 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:03.396640 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:03.396653 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:03.396677 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:03.396748 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:03.400999 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:03.401969 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:03.401980 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:03.401986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:03.402152 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:03.402602 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:03.402606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:03.402609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:03.403361 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:03.403709 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:03.404001 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 06:49:03.404472 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":496,"threads_started":1}
I20260807 06:49:03.413069 3704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38833 } }
I20260807 06:49:03.413432 3704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:03.413506 3704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea2a8bf125e421fa4c706b49ac1eb9e, State: Initialized, Role: FOLLOWER
I20260807 06:49:03.413985 3704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38833 } }
I20260807 06:49:03.414189 3704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:03.414285 3704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:03.414496 3704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:03.417003 3704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38833 } }
I20260807 06:49:03.417531 3704 leader_election.cc:302] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ea2a8bf125e421fa4c706b49ac1eb9e; no voters:
I20260807 06:49:03.418123 3704 leader_election.cc:288] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:03.418263 3713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:03.419062 3713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [term 1 LEADER]: Becoming Leader. State: Replica: 0ea2a8bf125e421fa4c706b49ac1eb9e, State: Running, Role: LEADER
I20260807 06:49:03.419699 3713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38833 } }
I20260807 06:49:03.420491 3704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:03.422753 3714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38833 } } }
I20260807 06:49:03.423218 3714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:03.422864 3717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea2a8bf125e421fa4c706b49ac1eb9e. Latest consensus state: current_term: 1 leader_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea2a8bf125e421fa4c706b49ac1eb9e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38833 } } }
I20260807 06:49:03.423655 3717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:03.425591 3728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:03.431931 3728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:03.440989 3728 catalog_manager.cc:1347] Generated new cluster ID: 545af6bf5c2a4cb19b4d4552c15eb36a
I20260807 06:49:03.441139 3728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:03.471809 3728 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:03.473847 3728 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:03.487397 3728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ea2a8bf125e421fa4c706b49ac1eb9e: Generated new TSK 0
I20260807 06:49:03.488517 3728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:49:03.562613 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38833
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:03.915185 3753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:03.915493 3753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:03.915680 3753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:03.923245 3753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:03.923393 3753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:03.923584 3753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:03.934015 3753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:03.935461 3753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:03.936830 3753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:03.950839 3758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:03.950578 3762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:03.950604 3765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:03.951938 3753 server_base.cc:1034] running on GCE node
I20260807 06:49:03.952577 3753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:03.953500 3753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:03.954789 3753 hybrid_clock.cc:648] HybridClock initialized: now 1786085343954737 us; error 49 us; skew 500 ppm
I20260807 06:49:03.955124 3753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:03 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085343, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:03.958407 3753 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:49:03.960038 3753 webserver.cc:460] Webserver started at http://127.27.54.65:40297/ using document root <none> and password file <none>
I20260807 06:49:03.960827 3753 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:03.960979 3753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:03.961308 3753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:03.964444 3753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "8d2a01d4de3148ef9a7ad6bfe4560b35"
format_stamp: "Formatted at 2026-08-07 06:49:03 on dist-test-slave-3kk6"
server_key: "40224ea47d6c3643cbc0b7e465db9044"
server_key_iv: "b9c011c16b9fe1b0ede89e47b64b6c74"
server_key_version: "encryptionkey@0"
I20260807 06:49:03.965287 3753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "8d2a01d4de3148ef9a7ad6bfe4560b35"
format_stamp: "Formatted at 2026-08-07 06:49:03 on dist-test-slave-3kk6"
server_key: "40224ea47d6c3643cbc0b7e465db9044"
server_key_iv: "b9c011c16b9fe1b0ede89e47b64b6c74"
server_key_version: "encryptionkey@0"
I20260807 06:49:03.971557 3753 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 06:49:03.975301 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:03.976763 3753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:03.976950 3753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "8d2a01d4de3148ef9a7ad6bfe4560b35"
format_stamp: "Formatted at 2026-08-07 06:49:03 on dist-test-slave-3kk6"
server_key: "40224ea47d6c3643cbc0b7e465db9044"
server_key_iv: "b9c011c16b9fe1b0ede89e47b64b6c74"
server_key_version: "encryptionkey@0"
I20260807 06:49:03.977175 3753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:03.996889 3753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:04.001348 3753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:04.001677 3753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:04.002796 3753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:04.004243 3753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:04.004350 3753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:04.004515 3753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:04.004598 3753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:04.040071 3753 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43169
I20260807 06:49:04.040122 3941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43169 every 8 connection(s)
I20260807 06:49:04.041528 3753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:04.042526 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 3753
I20260807 06:49:04.042817 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:04.043566 27865 external_mini_cluster.cc:1428] Setting key 6a08648e57461c69e1ea9dce4ff1ba6e
Aug 07 06:49:04 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085343, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:04.065580 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:04.044682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:48629 (local address 127.27.54.126:38833)
0807 06:49:04.045163 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:04.045168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:04.046418 (+ 1250us) server_negotiation.cc:407] Connection header received
0807 06:49:04.048423 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:04.048431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:04.048529 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:04.048657 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:04.050270 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.051438 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.053057 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.053791 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.053941 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:04.057949 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:04.057959 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:04.057966 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:04.058014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:04.061889 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:04.062734 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:04.062740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:04.062743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:04.062913 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:04.063364 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:04.063372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:04.063378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:04.063828 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:04.063985 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:04.064738 (+ 753us) server_negotiation.cc:299] Negotiation successful
0807 06:49:04.065034 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260807 06:49:04.066262 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:04.045343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38833 (local address 127.27.54.65:48629)
0807 06:49:04.046233 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:04.046291 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:04.048118 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:04.048905 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:04.048914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:04.049465 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:04.050073 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:04.050090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.051597 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.051601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:04.052895 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:04.052908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.054035 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.054039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:04.054248 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:04.055067 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:04.055096 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:04.057673 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:04.062069 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:04.062076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:04.062092 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:04.062563 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:04.063036 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:04.063039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:04.063041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:04.063237 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:04.063972 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:04.063979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:04.064344 (+ 365us) client_negotiation.cc:769] Sending connection context
0807 06:49:04.064598 (+ 254us) client_negotiation.cc:240] Negotiation successful
0807 06:49:04.065054 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":394,"threads_started":1}
I20260807 06:49:04.068101 3942 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38833
I20260807 06:49:04.068432 3942 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:04.069299 3942 heartbeater.cc:511] Master 127.27.54.126:38833 requested a full tablet report, sending...
I20260807 06:49:04.071899 3658 ts_manager.cc:194] Registered new tserver with Master: 8d2a01d4de3148ef9a7ad6bfe4560b35 (127.27.54.65:43169)
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
I20260807 06:49:04.074193 3658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:48629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:49:04.112039 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38833
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:04.467918 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:04.468217 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:04.468423 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:04.476337 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:04.476503 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:04.476689 3954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:04.487389 3954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:04.488878 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:04.490281 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:04.504177 3962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:04.504125 3963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:04.504200 3965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:04.505213 3954 server_base.cc:1034] running on GCE node
I20260807 06:49:04.505947 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:04.507047 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:04.508258 3954 hybrid_clock.cc:648] HybridClock initialized: now 1786085344508197 us; error 55 us; skew 500 ppm
I20260807 06:49:04.508605 3954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:04 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:04.512084 3954 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:49:04.513675 3954 webserver.cc:460] Webserver started at http://127.27.54.66:38531/ using document root <none> and password file <none>
I20260807 06:49:04.514403 3954 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:04.514554 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:04.514923 3954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:04.518036 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "e941bba22f2648cc831b585d2dc13a4d"
format_stamp: "Formatted at 2026-08-07 06:49:04 on dist-test-slave-3kk6"
server_key: "ceb1c41496f3110697b52777a037eac7"
server_key_iv: "114be6f589f5a6dbe2cc1f0b04d7de7c"
server_key_version: "encryptionkey@0"
I20260807 06:49:04.518970 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "e941bba22f2648cc831b585d2dc13a4d"
format_stamp: "Formatted at 2026-08-07 06:49:04 on dist-test-slave-3kk6"
server_key: "ceb1c41496f3110697b52777a037eac7"
server_key_iv: "114be6f589f5a6dbe2cc1f0b04d7de7c"
server_key_version: "encryptionkey@0"
I20260807 06:49:04.525341 3954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 06:49:04.529285 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:04.530746 3954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:49:04.530956 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "e941bba22f2648cc831b585d2dc13a4d"
format_stamp: "Formatted at 2026-08-07 06:49:04 on dist-test-slave-3kk6"
server_key: "ceb1c41496f3110697b52777a037eac7"
server_key_iv: "114be6f589f5a6dbe2cc1f0b04d7de7c"
server_key_version: "encryptionkey@0"
I20260807 06:49:04.531196 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:04.545352 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:04.549880 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:04.550215 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:04.551362 3954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:04.552825 3954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:04.552934 3954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:04.553089 3954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:04.553169 3954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:04.588277 3954 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:33375
I20260807 06:49:04.588320 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:33375 every 8 connection(s)
I20260807 06:49:04.589717 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:04.590261 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 3954
I20260807 06:49:04.590507 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:04.591137 27865 external_mini_cluster.cc:1428] Setting key e49bee3ebcd93b2cbd9f0d5d8a1dc0ed
Aug 07 06:49:04 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085344, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:04.611711 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:04.592940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:51455 (local address 127.27.54.126:38833)
0807 06:49:04.593442 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:04.593449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:04.594552 (+ 1103us) server_negotiation.cc:407] Connection header received
0807 06:49:04.596518 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:04.596524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:04.596617 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:04.596740 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:04.598320 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.599142 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.600769 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.601204 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.601350 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:04.605378 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:04.605383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:04.605386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:04.605421 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:04.608193 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:04.609206 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:04.609210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:04.609212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:04.609338 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:04.609909 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:04.609912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:04.609914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:04.610181 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:04.610295 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:04.611231 (+ 936us) server_negotiation.cc:299] Negotiation successful
0807 06:49:04.611411 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":242,"threads_started":1}
I20260807 06:49:04.613404 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:04.593590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38833 (local address 127.27.54.66:51455)
0807 06:49:04.594389 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:04.594440 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:04.596236 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:04.596996 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:04.597005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:04.597555 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:04.598139 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:04.598159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.599301 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.599308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:04.600596 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:04.600608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:04.601460 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:04.601468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:04.601704 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:04.602442 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:04.602466 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:04.605110 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:04.608360 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:04.608372 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:04.608392 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:04.609088 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:04.609464 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:04.609473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:04.609479 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:04.609797 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:04.610313 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:04.610324 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:04.610837 (+ 513us) client_negotiation.cc:769] Sending connection context
0807 06:49:04.611183 (+ 346us) client_negotiation.cc:240] Negotiation successful
0807 06:49:04.611841 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":340,"threads_started":1}
I20260807 06:49:04.614955 4158 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38833
I20260807 06:49:04.615298 4158 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:04.616103 4158 heartbeater.cc:511] Master 127.27.54.126:38833 requested a full tablet report, sending...
I20260807 06:49:04.617619 3658 ts_manager.cc:194] Registered new tserver with Master: e941bba22f2648cc831b585d2dc13a4d (127.27.54.66:33375)
I20260807 06:49:04.618443 3658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:51455
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:49:04.654165 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38833
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:05.006295 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:05.006610 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:05.006798 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:05.014289 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:05.014420 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:05.014539 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:05.025156 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:05.026479 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:05.027922 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:05.041807 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:05.041997 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:05.045593 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:05.047394 4168 server_base.cc:1034] running on GCE node
I20260807 06:49:05.048022 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:05.048976 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:05.050182 4168 hybrid_clock.cc:648] HybridClock initialized: now 1786085345050139 us; error 38 us; skew 500 ppm
I20260807 06:49:05.050515 4168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.053845 4168 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:49:05.056000 4168 webserver.cc:460] Webserver started at http://127.27.54.67:34327/ using document root <none> and password file <none>
I20260807 06:49:05.057120 4168 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:05.057292 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:05.057698 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:05.062572 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "d98aea09bc8b45e69fa389f9377e21ce"
format_stamp: "Formatted at 2026-08-07 06:49:05 on dist-test-slave-3kk6"
server_key: "b9acd090a076c0b96bc62a1a9a60f53e"
server_key_iv: "78d38d6db8a29a21d294cf5e3dd23fbb"
server_key_version: "encryptionkey@0"
I20260807 06:49:05.063908 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "d98aea09bc8b45e69fa389f9377e21ce"
format_stamp: "Formatted at 2026-08-07 06:49:05 on dist-test-slave-3kk6"
server_key: "b9acd090a076c0b96bc62a1a9a60f53e"
server_key_iv: "78d38d6db8a29a21d294cf5e3dd23fbb"
server_key_version: "encryptionkey@0"
I20260807 06:49:05.070194 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 06:49:05.073921 4190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:05.075359 4168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 06:49:05.075547 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "d98aea09bc8b45e69fa389f9377e21ce"
format_stamp: "Formatted at 2026-08-07 06:49:05 on dist-test-slave-3kk6"
server_key: "b9acd090a076c0b96bc62a1a9a60f53e"
server_key_iv: "78d38d6db8a29a21d294cf5e3dd23fbb"
server_key_version: "encryptionkey@0"
I20260807 06:49:05.075794 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:05.077778 3942 heartbeater.cc:503] Master 127.27.54.126:38833 was elected leader, sending a full tablet report...
I20260807 06:49:05.088555 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:05.092895 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:05.093216 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:05.094240 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:05.095839 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:05.095947 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:05.096102 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:05.096179 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:05.131215 4168 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:35619
I20260807 06:49:05.131254 4366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:35619 every 8 connection(s)
I20260807 06:49:05.132591 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:05.132946 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 4168
I20260807 06:49:05.133160 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:49:05.133788 27865 external_mini_cluster.cc:1428] Setting key 9386faba8a5cea9341ec0030b04adf14
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:05.158020 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.135503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:58339 (local address 127.27.54.126:38833)
0807 06:49:05.136053 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.136057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.137567 (+ 1510us) server_negotiation.cc:407] Connection header received
0807 06:49:05.139231 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.139235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.139321 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.139451 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.141148 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.142547 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.144810 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.145303 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.145454 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.151185 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.151191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.151196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.151240 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.154456 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.155444 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.155450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.155454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.155594 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.156058 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.156063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.156066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.156353 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.156520 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.157387 (+ 867us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.157615 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":300,"threads_started":1}
I20260807 06:49:05.159114 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.136288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38833 (local address 127.27.54.67:58339)
0807 06:49:05.137157 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:05.137226 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:05.138973 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:05.139739 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:05.139752 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:05.140394 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:05.140967 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:05.140984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.142798 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.142804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:05.144648 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:05.144661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.145745 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.145753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:05.145965 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.146762 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:05.146789 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:05.150968 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:05.154621 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:05.154633 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:05.154677 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:05.155230 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:05.155717 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:05.155723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:05.155726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:05.155931 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:05.156487 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:05.156495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:05.157000 (+ 505us) client_negotiation.cc:769] Sending connection context
0807 06:49:05.157298 (+ 298us) client_negotiation.cc:240] Negotiation successful
0807 06:49:05.157866 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":327,"threads_started":1}
I20260807 06:49:05.160704 4367 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38833
I20260807 06:49:05.161032 4367 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:05.162007 4367 heartbeater.cc:511] Master 127.27.54.126:38833 requested a full tablet report, sending...
I20260807 06:49:05.163689 3658 ts_manager.cc:194] Registered new tserver with Master: d98aea09bc8b45e69fa389f9377e21ce (127.27.54.67:35619)
I20260807 06:49:05.164564 3658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:58339
I20260807 06:49:05.175241 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/test-admin.keytab.
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.210160 27865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 06:49:05.216171 4382 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:05.213332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.27.54.65:43169)
0807 06:49:05.214115 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.214122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.214148 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:49:05.214214 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.214221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.214445 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.214629 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.215693 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37478: BlockingRecv error: recv got EOF from 127.0.0.1:37478 (error 108)
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":401,"threads_started":1}
W20260807 06:49:05.216323 27865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc' of type FILE
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.218084 27865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.219655 27865 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:49:05.236961 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.221505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.27.54.65:43169)
0807 06:49:05.221743 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.221752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.222045 (+ 293us) server_negotiation.cc:407] Connection header received
0807 06:49:05.222255 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.222261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.222342 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.222458 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.223585 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.224596 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.225902 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.226495 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.226648 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.230689 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.230707 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.230727 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.230776 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.233680 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.234480 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.234490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.234493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.234740 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.235219 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.235224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.235244 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.235718 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.235968 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.236397 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.236665 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 06:49:05.243105 4382 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:05.241377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.27.54.65:43169)
0807 06:49:05.241598 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.241603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.241769 (+ 166us) server_negotiation.cc:407] Connection header received
0807 06:49:05.241901 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.241904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.241961 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.242041 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.242840 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37494: BlockingRecv error: recv got EOF from 127.0.0.1:37494 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 06:49:05.243287 27865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc' of type FILE
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.244908 27865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.246340 27865 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:49:05.260871 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.247880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.27.54.65:43169)
0807 06:49:05.248100 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.248106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.248376 (+ 270us) server_negotiation.cc:407] Connection header received
0807 06:49:05.248580 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.248584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.248641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.248747 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.249714 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.250436 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.251752 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.252195 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.252341 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.255987 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.255992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.255996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.256042 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.258566 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.259226 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.259230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.259232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.259361 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.259700 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.259704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.259705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.259975 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.260189 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.260480 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.260646 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260807 06:49:05.265239 4382 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:05.263664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.27.54.65:43169)
0807 06:49:05.263870 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.263876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.263997 (+ 121us) server_negotiation.cc:407] Connection header received
0807 06:49:05.264117 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.264121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.264194 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.264271 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.264975 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37520: BlockingRecv error: recv got EOF from 127.0.0.1:37520 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 06:49:05.265379 27865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786085287136147-27865-0/krb5kdc/krb5cc' of type FILE
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.267009 27865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:05.268337 27865 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:49:05.282608 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.269640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.27.54.65:43169)
0807 06:49:05.269858 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.269865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.270032 (+ 167us) server_negotiation.cc:407] Connection header received
0807 06:49:05.270151 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.270154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.270206 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.270289 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.271286 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.272019 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.273182 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.273580 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.273673 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.277084 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.277089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.277092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.277122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.279821 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.280500 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.280504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.280507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.280697 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.281127 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.281131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.281133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.281430 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.281701 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.282073 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.282315 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 06:49:05.284269 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 3753
I20260807 06:49:05.296305 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.284967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.27.54.65:43169)
0807 06:49:05.285115 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.285118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.285455 (+ 337us) server_negotiation.cc:407] Connection header received
0807 06:49:05.285559 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.285563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.285612 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.285687 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.286576 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.287311 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.288463 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.288879 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.288973 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.291339 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.291344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.291347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.291376 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.293788 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.294504 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.294509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.294511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.294751 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.295093 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.295097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.295098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.295384 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.295590 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.295845 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.296016 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:49:05.296918 3936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:05.383440 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 3753
I20260807 06:49:05.407692 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 3954
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:49:05.423722 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.408662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.27.54.66:33375)
0807 06:49:05.409251 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.409258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.409284 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:49:05.409350 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.409356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.409567 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.409705 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.410796 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.411854 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.413142 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.413705 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.413807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.418058 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.418071 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.418086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.418125 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.420938 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.421620 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.421627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.421629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.421774 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.422139 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.422142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.422144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.422506 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.422965 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.423095 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.423370 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":312,"threads_started":1}
I20260807 06:49:05.425058 4148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:05.509835 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 3954
I20260807 06:49:05.535089 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 4168
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[3588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:49:05.553398 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:05.536305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.27.54.67:35619)
0807 06:49:05.537054 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:05.537061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:05.537090 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:49:05.537190 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:05.537200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:05.537432 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:05.537608 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:05.538881 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.540295 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.541627 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:05.542233 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:05.542391 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:05.547679 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:05.547693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:05.547708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:05.547748 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:05.550562 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.551408 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.551416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.551419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.551566 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:05.551997 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:05.552000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:05.552002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:05.552372 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:05.552615 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:05.552874 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 06:49:05.553089 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":413,"threads_started":1}
I20260807 06:49:05.554983 4360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:05.641144 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 4168
I20260807 06:49:05.664682 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 3604
I20260807 06:49:05.665995 3698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:05.772061 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 3604
2026-08-07T06:49:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4455 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[4409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[4409](info): set up 2 sockets
Aug 07 06:49:05 dist-test-slave-3kk6 krb5kdc[4409](info): commencing operation
krb5kdc: starting...
W20260807 06:49:07.099193 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:49:07 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab.
Entry for principal oryx/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab.
I20260807 06:49:07.390020 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:42485
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab
--principal=oryx/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:42485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:07.741590 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:07.741943 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:07.742046 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:07.749405 4432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:07.749536 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:07.749586 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:07.749639 4432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:07.749684 4432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:07.760025 4432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:42485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab
--principal=oryx/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:07.761412 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:07.762794 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:07.774955 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:07.775076 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:07.775048 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:07.776010 4432 server_base.cc:1034] running on GCE node
I20260807 06:49:07.776923 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:07.778427 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:07.779667 4432 hybrid_clock.cc:648] HybridClock initialized: now 1786085347779631 us; error 56 us; skew 500 ppm
I20260807 06:49:07.780033 4432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:07 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:07.783327 4432 init.cc:377] Logged in from keytab as oryx/127.27.54.126@KRBTEST.COM (short username oryx)
I20260807 06:49:07.784936 4432 webserver.cc:460] Webserver started at http://127.27.54.126:44025/ using document root <none> and password file <none>
I20260807 06:49:07.785768 4432 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:07.785928 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:07.786255 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:07.789397 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "d8d3f22699b24f46b6bf3d716f39b696"
format_stamp: "Formatted at 2026-08-07 06:49:07 on dist-test-slave-3kk6"
server_key: "9f26db35237afa5a7741e438d725bf21"
server_key_iv: "3493b740821e7e458881dbe0924866d2"
server_key_version: "encryptionkey@0"
I20260807 06:49:07.790216 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "d8d3f22699b24f46b6bf3d716f39b696"
format_stamp: "Formatted at 2026-08-07 06:49:07 on dist-test-slave-3kk6"
server_key: "9f26db35237afa5a7741e438d725bf21"
server_key_iv: "3493b740821e7e458881dbe0924866d2"
server_key_version: "encryptionkey@0"
I20260807 06:49:07.795631 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 06:49:07.799206 4450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:07.800536 4432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 06:49:07.800731 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "d8d3f22699b24f46b6bf3d716f39b696"
format_stamp: "Formatted at 2026-08-07 06:49:07 on dist-test-slave-3kk6"
server_key: "9f26db35237afa5a7741e438d725bf21"
server_key_iv: "3493b740821e7e458881dbe0924866d2"
server_key_version: "encryptionkey@0"
I20260807 06:49:07.800958 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:07.812685 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:07.816865 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:07.817170 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:07.836545 4432 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:42485
I20260807 06:49:07.836578 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:42485 every 8 connection(s)
I20260807 06:49:07.838208 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:07.842042 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:07.848582 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 4432
I20260807 06:49:07.848870 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:07.849530 27865 external_mini_cluster.cc:1428] Setting key b50cf11f0950d0705d6bce12fd0f950b
I20260807 06:49:07.853986 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Bootstrap starting.
I20260807 06:49:07.858160 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:07.859558 4534 log.cc:824] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:07.863157 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: No bootstrap required, opened a new log
Aug 07 06:49:07 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:07.872967 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } }
I20260807 06:49:07.873432 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:07.873569 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d3f22699b24f46b6bf3d716f39b696, State: Initialized, Role: FOLLOWER
I20260807 06:49:07.874357 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } }
I20260807 06:49:07.874643 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:07.874843 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:07.875044 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:07.875916 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:07.851941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.27.54.126:42485)
0807 06:49:07.852798 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:07.852814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:07.852866 (+ 52us) server_negotiation.cc:407] Connection header received
0807 06:49:07.855354 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:07.855384 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:07.855928 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:07.856449 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:07.857856 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:07.859956 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:07.861683 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:07.862777 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:07.862959 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:07.867563 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:07.867571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:07.867586 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:07.867627 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:07.871017 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:07.872001 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:07.872007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:07.872015 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:07.872194 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:07.872657 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:07.872660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:07.872662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:07.873217 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:07.873597 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:07.874097 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 06:49:07.874585 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":373,"threads_started":1}
I20260807 06:49:07.877689 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } }
I20260807 06:49:07.878216 4534 leader_election.cc:302] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d3f22699b24f46b6bf3d716f39b696; no voters:
I20260807 06:49:07.879035 4534 leader_election.cc:288] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:07.879141 4543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:07.879894 4543 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 LEADER]: Becoming Leader. State: Replica: d8d3f22699b24f46b6bf3d716f39b696, State: Running, Role: LEADER
I20260807 06:49:07.880489 4543 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } }
I20260807 06:49:07.881203 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:07.883598 4544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } }
I20260807 06:49:07.883668 4545 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d3f22699b24f46b6bf3d716f39b696. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } }
I20260807 06:49:07.884068 4544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:07.884115 4545 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:07.887738 4550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:07.893751 4550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:07.902985 4550 catalog_manager.cc:1347] Generated new cluster ID: b4ece8fc898c4f84b0e66249daa1d237
I20260807 06:49:07.903131 4550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:07.911391 4550 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:07.913244 4550 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:07.926728 4550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Generated new TSK 0
I20260807 06:49:07.927857 4550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab.
Entry for principal oryx/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab.
I20260807 06:49:08.003363 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42485
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:08.382503 4569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:08.382910 4569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:08.383018 4569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:08.390769 4569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:08.390894 4569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:08.391012 4569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:08.401304 4569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:08.402719 4569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:08.404135 4569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:08.418399 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:08.418401 4578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:08.418399 4583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:08.419379 4569 server_base.cc:1034] running on GCE node
I20260807 06:49:08.420059 4569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:08.421101 4569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:08.422330 4569 hybrid_clock.cc:648] HybridClock initialized: now 1786085348422274 us; error 49 us; skew 500 ppm
I20260807 06:49:08.422753 4569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:08 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085348, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:08.426196 4569 init.cc:377] Logged in from keytab as oryx/127.27.54.65@KRBTEST.COM (short username oryx)
I20260807 06:49:08.427852 4569 webserver.cc:460] Webserver started at http://127.27.54.65:36719/ using document root <none> and password file <none>
I20260807 06:49:08.428614 4569 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:08.428757 4569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:08.429111 4569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:08.432485 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "12c49bf3489f4112ab6bde0f7612c3b9"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "072bd4b573bccf5e5853cab1f75ee15a"
server_key_iv: "a8fe649914a8311b9afe660d21ad2d0a"
server_key_version: "encryptionkey@0"
I20260807 06:49:08.433367 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "12c49bf3489f4112ab6bde0f7612c3b9"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "072bd4b573bccf5e5853cab1f75ee15a"
server_key_iv: "a8fe649914a8311b9afe660d21ad2d0a"
server_key_version: "encryptionkey@0"
I20260807 06:49:08.439007 4569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:49:08.442574 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:08.444036 4569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:08.444212 4569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "12c49bf3489f4112ab6bde0f7612c3b9"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "072bd4b573bccf5e5853cab1f75ee15a"
server_key_iv: "a8fe649914a8311b9afe660d21ad2d0a"
server_key_version: "encryptionkey@0"
I20260807 06:49:08.444446 4569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:08.470515 4569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:08.474924 4569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:08.475255 4569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:08.476358 4569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:08.477880 4569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:08.477962 4569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:08.478118 4569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:08.478188 4569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:08.514895 4569 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:35161
I20260807 06:49:08.514947 4762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:35161 every 8 connection(s)
I20260807 06:49:08.516424 4569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:08.526239 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 4569
I20260807 06:49:08.526512 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:08.527259 27865 external_mini_cluster.cc:1428] Setting key 2d01fe9f5996e5747279e09bdd74cb70
Aug 07 06:49:08 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085348, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:08.540721 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:08.519436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:55419 (local address 127.27.54.126:42485)
0807 06:49:08.519987 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:08.519992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:08.521031 (+ 1039us) server_negotiation.cc:407] Connection header received
0807 06:49:08.523129 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:08.523136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:08.523243 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:08.523375 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:08.525056 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:08.525982 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:08.527972 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:08.528578 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:08.528699 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:08.532804 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:08.532815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:08.532823 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:08.532874 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:08.536778 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:08.537711 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:08.537719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:08.537725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:08.537900 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:08.538480 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:08.538488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:08.538494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:08.538947 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:08.539119 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:08.539924 (+ 805us) server_negotiation.cc:299] Negotiation successful
0807 06:49:08.540219 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":289,"threads_started":1}
I20260807 06:49:08.541492 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:08.520021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:55419)
0807 06:49:08.520830 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:08.520882 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:08.522813 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:08.523649 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:08.523658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:08.524231 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:08.524846 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:08.524865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:08.526185 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:08.526189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:08.527757 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:08.527772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:08.528853 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:08.528862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:08.529173 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:08.530064 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:08.530088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:08.532516 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:08.536993 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:08.537001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:08.537018 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:08.537535 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:08.538065 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:08.538069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:08.538072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:08.538332 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:08.539106 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:08.539114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:08.539484 (+ 370us) client_negotiation.cc:769] Sending connection context
0807 06:49:08.539753 (+ 269us) client_negotiation.cc:240] Negotiation successful
0807 06:49:08.540205 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":316,"threads_started":1}
I20260807 06:49:08.543403 4763 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42485
I20260807 06:49:08.543768 4763 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:08.544732 4763 heartbeater.cc:511] Master 127.27.54.126:42485 requested a full tablet report, sending...
I20260807 06:49:08.547338 4480 ts_manager.cc:194] Registered new tserver with Master: 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:08.549118 4480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:55419
WARNING: no policy specified for oryx/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab.
Entry for principal oryx/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab.
I20260807 06:49:08.595417 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42485
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:08.952843 4771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:08.953153 4771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:08.953329 4771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:08.961205 4771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:08.961334 4771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:08.961513 4771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:08.971990 4771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:08.973407 4771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:08.974844 4771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:08.988907 4781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:08.989953 4780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:08.990219 4785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:08.990635 4771 server_base.cc:1034] running on GCE node
I20260807 06:49:08.991423 4771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:08.992479 4771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:08.993718 4771 hybrid_clock.cc:648] HybridClock initialized: now 1786085348993663 us; error 65 us; skew 500 ppm
I20260807 06:49:08.994058 4771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:08 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085348, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:08.997548 4771 init.cc:377] Logged in from keytab as oryx/127.27.54.66@KRBTEST.COM (short username oryx)
I20260807 06:49:08.999262 4771 webserver.cc:460] Webserver started at http://127.27.54.66:45763/ using document root <none> and password file <none>
I20260807 06:49:08.999992 4771 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:09.000154 4771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:09.000499 4771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:09.003791 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "5ec855a623ec4065b270632de521ca74"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "0e13d4f24d3c0f04fd417fc63f42458a"
server_key_iv: "201e3d1cacd123077b250e0dfb22ca1f"
server_key_version: "encryptionkey@0"
I20260807 06:49:09.004736 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "5ec855a623ec4065b270632de521ca74"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "0e13d4f24d3c0f04fd417fc63f42458a"
server_key_iv: "201e3d1cacd123077b250e0dfb22ca1f"
server_key_version: "encryptionkey@0"
I20260807 06:49:09.010500 4771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:49:09.014426 4793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:09.016014 4771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:09.016230 4771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "5ec855a623ec4065b270632de521ca74"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "0e13d4f24d3c0f04fd417fc63f42458a"
server_key_iv: "201e3d1cacd123077b250e0dfb22ca1f"
server_key_version: "encryptionkey@0"
I20260807 06:49:09.016464 4771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:09.031324 4771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:09.035799 4771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:09.036134 4771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:09.037181 4771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:09.038740 4771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:09.038852 4771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:09.039019 4771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:09.039103 4771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:09.075793 4771 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:36113
I20260807 06:49:09.075843 4967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:36113 every 8 connection(s)
I20260807 06:49:09.077157 4771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:09.085321 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 4771
I20260807 06:49:09.085572 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:09.086126 27865 external_mini_cluster.cc:1428] Setting key 2439fed86716252ed76b55ec15686fa0
Aug 07 06:49:09 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085348, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:09.102181 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:09.079983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:46427 (local address 127.27.54.126:42485)
0807 06:49:09.080452 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:09.080457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:09.081600 (+ 1143us) server_negotiation.cc:407] Connection header received
0807 06:49:09.083687 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:09.083693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:09.083795 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:09.083922 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:09.085964 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.086983 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.089072 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.089843 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.089997 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:09.093939 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:09.093950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:09.093958 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:09.094008 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:09.098121 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:09.099177 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:09.099186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:09.099192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:09.099397 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:09.099946 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:09.099953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:09.099959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:09.100401 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:09.100572 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:09.101659 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0807 06:49:09.101890 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260807 06:49:09.103876 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:09.080574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:46427)
0807 06:49:09.081332 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:09.081424 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:09.083397 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:09.084173 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:09.084182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:09.084757 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:09.085708 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:09.085739 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.087183 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.087188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:09.088906 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:09.088919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.090111 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.090118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:09.090356 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:09.091239 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:09.091264 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:09.093646 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:09.098352 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:09.098361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:09.098378 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:09.099022 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:09.099544 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:09.099548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:09.099550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:09.099806 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:09.100590 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:09.100602 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:09.101141 (+ 539us) client_negotiation.cc:769] Sending connection context
0807 06:49:09.101520 (+ 379us) client_negotiation.cc:240] Negotiation successful
0807 06:49:09.102207 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":311,"threads_started":1}
I20260807 06:49:09.105742 4973 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42485
I20260807 06:49:09.106187 4973 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:09.107103 4973 heartbeater.cc:511] Master 127.27.54.126:42485 requested a full tablet report, sending...
I20260807 06:49:09.108759 4480 ts_manager.cc:194] Registered new tserver with Master: 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113)
I20260807 06:49:09.109612 4480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:46427
WARNING: no policy specified for oryx/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab.
Entry for principal oryx/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab.
I20260807 06:49:09.153725 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42485
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:09.509987 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:09.510303 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:09.510499 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:09.518136 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:09.518258 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:09.518446 4980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:09.528680 4980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:09.530061 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:09.531641 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:09.546002 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:09.546074 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:09.545977 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:09.547276 4980 server_base.cc:1034] running on GCE node
I20260807 06:49:09.548044 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:09.549170 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:09.550437 4980 hybrid_clock.cc:648] HybridClock initialized: now 1786085349550365 us; error 45 us; skew 500 ppm
I20260807 06:49:09.550835 4980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:09.552597 4763 heartbeater.cc:503] Master 127.27.54.126:42485 was elected leader, sending a full tablet report...
Aug 07 06:49:09 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085349, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:09.555009 4980 init.cc:377] Logged in from keytab as oryx/127.27.54.67@KRBTEST.COM (short username oryx)
I20260807 06:49:09.556596 4980 webserver.cc:460] Webserver started at http://127.27.54.67:37455/ using document root <none> and password file <none>
I20260807 06:49:09.557300 4980 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:09.557413 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:09.557686 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:09.560962 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "b5b11280d810438b9c5efb9800a5e40a"
format_stamp: "Formatted at 2026-08-07 06:49:09 on dist-test-slave-3kk6"
server_key: "321a5e26bca4d4f1ad903f2b132514c5"
server_key_iv: "11af0ac5b9e0f3bf0914af1c99b14196"
server_key_version: "encryptionkey@0"
I20260807 06:49:09.561887 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "b5b11280d810438b9c5efb9800a5e40a"
format_stamp: "Formatted at 2026-08-07 06:49:09 on dist-test-slave-3kk6"
server_key: "321a5e26bca4d4f1ad903f2b132514c5"
server_key_iv: "11af0ac5b9e0f3bf0914af1c99b14196"
server_key_version: "encryptionkey@0"
I20260807 06:49:09.567708 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:49:09.571789 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:09.573473 4980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:49:09.573639 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "b5b11280d810438b9c5efb9800a5e40a"
format_stamp: "Formatted at 2026-08-07 06:49:09 on dist-test-slave-3kk6"
server_key: "321a5e26bca4d4f1ad903f2b132514c5"
server_key_iv: "11af0ac5b9e0f3bf0914af1c99b14196"
server_key_version: "encryptionkey@0"
I20260807 06:49:09.573843 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:09.592156 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:09.597306 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:09.597594 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:09.598627 4980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:09.600169 4980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:09.600247 4980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:09.600337 4980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:09.600380 4980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:09.636835 4980 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:44931
I20260807 06:49:09.636881 5168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:44931 every 8 connection(s)
I20260807 06:49:09.638305 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:09.643757 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 4980
I20260807 06:49:09.644021 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:49:09.644502 27865 external_mini_cluster.cc:1428] Setting key 1830740c968efedb87ba1501390f3eef
Aug 07 06:49:09 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085349, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:09.661693 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:09.641207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:55909 (local address 127.27.54.126:42485)
0807 06:49:09.641726 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:09.641733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:09.642862 (+ 1129us) server_negotiation.cc:407] Connection header received
0807 06:49:09.645024 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:09.645031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:09.645133 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:09.645241 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:09.647380 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.648564 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.650382 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.650883 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.651000 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:09.654857 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:09.654863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:09.654867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:09.654907 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:09.657786 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:09.658712 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:09.658718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:09.658721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:09.658928 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:09.659498 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:09.659502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:09.659504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:09.659955 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:09.660132 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:09.661116 (+ 984us) server_negotiation.cc:299] Negotiation successful
0807 06:49:09.661398 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":229,"threads_started":1}
I20260807 06:49:09.662868 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:09.641826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:55909)
0807 06:49:09.642604 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:09.642712 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:09.644686 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:09.645568 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:09.645578 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:09.646268 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:09.647145 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:09.647178 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.648738 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.648743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:09.650194 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:09.650210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:09.651125 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:09.651130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:09.651371 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:09.652131 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:09.652157 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:09.654544 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:09.657981 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:09.657989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:09.658005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:09.658507 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:09.659066 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:09.659071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:09.659076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:09.659348 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:09.660107 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:09.660113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:09.660632 (+ 519us) client_negotiation.cc:769] Sending connection context
0807 06:49:09.661013 (+ 381us) client_negotiation.cc:240] Negotiation successful
0807 06:49:09.661536 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":300,"threads_started":1}
I20260807 06:49:09.664443 5170 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42485
I20260807 06:49:09.664733 5170 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:09.665518 5170 heartbeater.cc:511] Master 127.27.54.126:42485 requested a full tablet report, sending...
I20260807 06:49:09.667112 4480 ts_manager.cc:194] Registered new tserver with Master: b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931)
I20260807 06:49:09.668015 4480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:55909
I20260807 06:49:09.674059 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.54.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.125.keytab.
Entry for principal oryx/127.27.54.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.125.keytab.
I20260807 06:49:09.743449 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:44405
--webserver_interface=127.27.54.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.125.keytab
--principal=oryx/127.27.54.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:42485,127.27.54.125:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:10.112640 4973 heartbeater.cc:503] Master 127.27.54.126:42485 was elected leader, sending a full tablet report...
W20260807 06:49:10.120785 5177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:10.121192 5177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:10.121354 5177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:10.129201 5177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:10.129403 5177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:10.129521 5177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:10.129628 5177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:10.129714 5177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:10.140411 5177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:42485,127.27.54.125:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.125.keytab
--principal=oryx/127.27.54.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.54.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
W20260807 06:49:10.140887 5177 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.54.126:42485,127.27.54.125:44405'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 06:49:10.142045 5177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:10.143440 5177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:10.156584 5187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:10.156991 5186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:10.156646 5190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:10.157577 5177 server_base.cc:1034] running on GCE node
I20260807 06:49:10.158524 5177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:10.160187 5177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:10.161438 5177 hybrid_clock.cc:648] HybridClock initialized: now 1786085350161378 us; error 43 us; skew 500 ppm
I20260807 06:49:10.161821 5177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:10.165519 5177 init.cc:377] Logged in from keytab as oryx/127.27.54.125@KRBTEST.COM (short username oryx)
I20260807 06:49:10.167421 5177 webserver.cc:460] Webserver started at http://127.27.54.125:42561/ using document root <none> and password file <none>
I20260807 06:49:10.168246 5177 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:10.168406 5177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:10.168767 5177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:10.172233 5177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/instance:
uuid: "02ef9d212a134bf88539c91ff2ea6089"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "c74bbbed0907f5453e013341adb5f06c"
server_key_iv: "bc4a14c7eb164d9709909160ad5bebb9"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.173168 5177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal/instance:
uuid: "02ef9d212a134bf88539c91ff2ea6089"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "c74bbbed0907f5453e013341adb5f06c"
server_key_iv: "bc4a14c7eb164d9709909160ad5bebb9"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.179502 5177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 06:49:10.183645 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:10.185328 5177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:10.185545 5177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
uuid: "02ef9d212a134bf88539c91ff2ea6089"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "c74bbbed0907f5453e013341adb5f06c"
server_key_iv: "bc4a14c7eb164d9709909160ad5bebb9"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.185798 5177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:10.205133 5177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:10.210055 5177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:10.210386 5177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:10.231482 5177 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.125:44405
I20260807 06:49:10.231506 5271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.125:44405 every 8 connection(s)
I20260807 06:49:10.233245 5177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
I20260807 06:49:10.234295 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 5177
I20260807 06:49:10.238155 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:10.245836 5273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.125@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:10.260293 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.236976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.27.54.125:44405)
0807 06:49:10.237856 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.237870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.237912 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:49:10.239730 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.239753 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.240146 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.240579 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.242273 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.243823 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.245200 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.245993 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.246241 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.251896 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.251905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.251916 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.251955 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.255269 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.256086 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.256093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.256100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.256314 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.256785 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.256792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.256798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.257374 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.257706 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.258262 (+ 556us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.258726 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":385,"threads_started":1}
I20260807 06:49:10.265355 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.249017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.27.54.126:42485)
0807 06:49:10.249595 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.249603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.250225 (+ 622us) server_negotiation.cc:407] Connection header received
0807 06:49:10.250508 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.250513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.250583 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.250790 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.251990 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.252740 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.254275 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.254763 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.254915 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.258589 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.258599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.258607 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.258715 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.262102 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.263040 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.263046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.263049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.263221 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.263736 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.263740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.263742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.264076 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.264242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.264741 (+ 499us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.264972 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":309,"threads_started":1}
I20260807 06:49:10.265391 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.249025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42314)
0807 06:49:10.250012 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.250079 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.250307 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.250865 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.250871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.251499 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:10.251804 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.251815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.252924 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.252929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.254112 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.254125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.254997 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.255003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.255183 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.255847 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:10.255864 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:10.258346 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:10.262285 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.262295 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.262300 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.262896 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.263378 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.263386 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.263392 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.263575 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.264234 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:10.264241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:10.264410 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.264597 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.264958 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":554,"threads_started":1}
I20260807 06:49:10.267280 5273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.125@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
I20260807 06:49:10.280366 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.268172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43624)
0807 06:49:10.268340 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.268361 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.268506 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.268835 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.268839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.269225 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:10.269361 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.269370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.270370 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.270375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.271321 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.271333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.272042 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.272047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.272273 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.272853 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:10.272870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:10.275014 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:10.278060 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.278063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.278066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.278507 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.278937 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.278941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.278943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.279074 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.279596 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:10.279600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:10.279734 (+ 134us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.279850 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.280079 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":44}
I20260807 06:49:10.280671 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.268096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.27.54.125:44405)
0807 06:49:10.268338 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.268343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.268444 (+ 101us) server_negotiation.cc:407] Connection header received
0807 06:49:10.268633 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.268644 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.268697 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.268820 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.269497 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.270210 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.271476 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.271890 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.272029 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.275212 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.275224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.275227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.275259 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.277924 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.278601 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.278606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.278608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.278823 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.279176 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.279179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.279181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.279491 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.279606 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.279941 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.280246 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":55}
I20260807 06:49:10.285790 5273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Bootstrap starting.
I20260807 06:49:10.289441 5273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:10.290632 5273 log.cc:824] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:10.293757 5273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: No bootstrap required, opened a new log
I20260807 06:49:10.301256 5273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } }
I20260807 06:49:10.301614 5273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:10.301697 5273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ef9d212a134bf88539c91ff2ea6089, State: Initialized, Role: FOLLOWER
I20260807 06:49:10.302250 5273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } }
I20260807 06:49:10.303166 5284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } } }
I20260807 06:49:10.303484 5284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:10.304318 5273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:49:10.314175 5302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:49:10.314445 5302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:49:10.318800 5177 master_runner.cc:428] Detected that this master 02ef9d212a134bf88539c91ff2ea6089 is joining an existing cluster
I20260807 06:49:10.318944 5177 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.54.125:44405
I20260807 06:49:10.344147 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.325990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.27.54.126:42485)
0807 06:49:10.326200 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.326207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.326232 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:49:10.326316 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.326323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.326460 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.326571 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.328525 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.329809 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.333366 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.334082 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.334249 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.336786 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.336794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.336799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.336844 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.340645 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.341546 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.341552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.341555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.341718 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.342183 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.342188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.342191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.342599 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.342787 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.343508 (+ 721us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.343789 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:49:10.344149 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.325542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43628)
0807 06:49:10.326313 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.326333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.326463 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.326832 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.326837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.328192 (+ 1355us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:10.328600 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.328614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.330092 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.330097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.331733 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.331759 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.332931 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.332942 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.333206 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.333994 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:10.334018 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:10.335351 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:10.340681 (+ 5330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.340689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.340697 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.341371 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.341837 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.341842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.341845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.342044 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.342752 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:10.342757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:10.342936 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.343189 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.343549 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":118,"spinlock_wait_cycles":29568,"thread_start_us":1645,"threads_started":1}
I20260807 06:49:10.345499 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.326256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.27.54.125:44405)
0807 06:49:10.326463 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.326470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.326492 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:49:10.326565 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.326571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.326684 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.326804 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.328783 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.329865 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.331929 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.332670 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.332834 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.335572 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.335583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.335588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.335628 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.340496 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.341526 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.341531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.341534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.341709 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.342186 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.342191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.342194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.342582 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.344689 (+ 2107us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.344835 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.345099 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:49:10.345938 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.324335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42324)
0807 06:49:10.325165 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.325191 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.325494 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.326832 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.326837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.327737 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:10.328287 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.328305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.330035 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.330042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.333198 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.333209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.334323 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.334329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.334516 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.335216 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:10.335234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:10.336489 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:10.340864 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.340869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.340873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.341407 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.341854 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.341858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.341860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.342050 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.342843 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:10.342847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:10.343025 (+ 178us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.343191 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.345272 (+ 2081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":1808,"spinlock_wait_cycles":20736,"thread_start_us":2564,"threads_started":1}
WARNING: no policy specified for oryx/127.27.54.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.124@KRBTEST.COM" created.
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.126@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
I20260807 06:49:10.372998 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.353601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.27.54.125:44405)
0807 06:49:10.354608 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.354615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.354634 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:49:10.354766 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.354770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.354869 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.354997 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.356200 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.356992 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.358723 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.359237 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.359434 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.364232 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.364241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.364249 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.364298 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.368677 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.369997 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.370006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.370016 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.370225 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.370810 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.370818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.370823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.371324 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.371482 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.372218 (+ 736us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.372516 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855}
I20260807 06:49:10.373031 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.353474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43640)
0807 06:49:10.354194 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.354236 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.354460 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.355062 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.355070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.355773 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:10.356006 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.356018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.357147 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.357155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.358442 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.358474 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.359472 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.359479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.359729 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.360594 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:10.360650 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:10.363949 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:10.368945 (+ 4996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.368968 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.368980 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.369800 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.370399 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.370407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.370413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.370617 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.371523 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:10.371535 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:10.371800 (+ 265us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.372061 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.372545 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260807 06:49:10.375907 4480 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.54.125:44405: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089"
member_type: NON_VOTER
last_known_addr {
host: "127.27.54.125"
port: 44405
}
attrs {
promote: true
}
}
dest_uuid: "d8d3f22699b24f46b6bf3d716f39b696"
cas_config_opid_index: -1
I20260807 06:49:10.377640 4480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } }
I20260807 06:49:10.393978 5236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:10.393501 5320 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } }
I20260807 06:49:10.398485 5236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 06:49:10.399437 5320 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.54.125:44405
I20260807 06:49:10.400283 5327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:10.400681 5177 master.cc:537] Master@127.27.54.125:44405 shutting down...
I20260807 06:49:10.401031 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } }
I20260807 06:49:10.401503 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.124.keytab.
Entry for principal oryx/127.27.54.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.124.keytab.
W20260807 06:49:10.407104 4452 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260807 06:49:10.407346 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:38271
--webserver_interface=127.27.54.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.124.keytab
--principal=oryx/127.27.54.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:10.423849 5177 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:10.424444 5177 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 06:49:10.424608 4452 consensus_peers.cc:487] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 -> Peer 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125:44405): Couldn't send request to peer 02ef9d212a134bf88539c91ff2ea6089. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 06:49:10.424734 5177 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: stopping tablet replica
I20260807 06:49:10.431711 5177 master.cc:552] Master@127.27.54.125:44405 shutdown complete.
I20260807 06:49:10.431885 5177 master_runner.cc:305] Clearing existing system tablet
I20260807 06:49:10.432353 5177 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:10.432485 5177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:10.435042 5177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260807 06:49:10.436679 5333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:10.437065 5177 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:10.437390 5177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 06:49:10.437520 5177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
uuid: "02ef9d212a134bf88539c91ff2ea6089"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "c74bbbed0907f5453e013341adb5f06c"
server_key_iv: "bc4a14c7eb164d9709909160ad5bebb9"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.441637 5177 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:10.446749 5177 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 06:49:10.446971 5177 log.cc:1196] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 06:49:10.447506 5177 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Deleting consensus metadata
I20260807 06:49:10.447785 5177 master_runner.cc:315] Copying system tablet from 127.27.54.126:42485
I20260807 06:49:10.451481 5177 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: tablet copy: Beginning tablet copy session from remote peer at address 127.27.54.126:42485
I20260807 06:49:10.473268 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.452638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.27.54.126:42485)
0807 06:49:10.452893 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.452902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.459957 (+ 7055us) server_negotiation.cc:407] Connection header received
0807 06:49:10.460226 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.460231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.460310 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.460430 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:10.461364 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.462124 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.464098 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.464782 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.464870 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.467164 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:10.467169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:10.467173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:10.467214 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:10.470196 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.471046 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.471059 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.471061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.471203 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:10.471661 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:10.471665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:10.471668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:10.471983 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:10.472161 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.472594 (+ 433us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.472840 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 06:49:10.473295 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.452599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42330)
0807 06:49:10.459804 (+ 7205us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.459825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.460044 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.460476 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.460481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.460935 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:10.461179 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.461192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.462318 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.462323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.463872 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.463894 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.465120 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.465124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.465257 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.465993 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:10.466011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:10.466922 (+ 911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:10.470374 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.470380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.470385 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.470898 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.471333 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:10.471338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:10.471341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:10.471502 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:10.472124 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:10.472128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:10.472323 (+ 195us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.472480 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.472841 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7028,"thread_start_us":6896,"threads_started":1}
I20260807 06:49:10.474925 4507 tablet_copy_service.cc:140] P d8d3f22699b24f46b6bf3d716f39b696: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 02ef9d212a134bf88539c91ff2ea6089 ({username='oryx', principal='oryx/127.27.54.125@KRBTEST.COM'} at 127.0.0.1:42330)
I20260807 06:49:10.475347 4507 tablet_copy_service.cc:161] P d8d3f22699b24f46b6bf3d716f39b696: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 02ef9d212a134bf88539c91ff2ea6089 at {username='oryx', principal='oryx/127.27.54.125@KRBTEST.COM'} at 127.0.0.1:42330: session id = 02ef9d212a134bf88539c91ff2ea6089-00000000000000000000000000000000
I20260807 06:49:10.479290 4507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 06:49:10.482007 5177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 06:49:10.487591 5177 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: tablet copy: Starting download of 0 data blocks...
I20260807 06:49:10.488072 5177 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: tablet copy: Starting download of 1 WAL segments...
I20260807 06:49:10.491937 5177 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 06:49:10.495882 4507 tablet_copy_service.cc:342] P d8d3f22699b24f46b6bf3d716f39b696: Request end of tablet copy session 02ef9d212a134bf88539c91ff2ea6089-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.54.125@KRBTEST.COM'} at 127.0.0.1:42330
I20260807 06:49:10.496191 4507 tablet_copy_service.cc:433] P d8d3f22699b24f46b6bf3d716f39b696: ending tablet copy session 02ef9d212a134bf88539c91ff2ea6089-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 02ef9d212a134bf88539c91ff2ea6089
I20260807 06:49:10.510959 5177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:10.514317 5342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:10.519150 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:10.522470 5177 server_base.cc:1034] running on GCE node
W20260807 06:49:10.522893 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:10.523445 5177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:10.524236 5177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:10.525549 5177 hybrid_clock.cc:648] HybridClock initialized: now 1786085350525477 us; error 41 us; skew 500 ppm
I20260807 06:49:10.525837 5177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:10.528558 5177 init.cc:377] Logged in from keytab as oryx/127.27.54.125@KRBTEST.COM (short username oryx)
I20260807 06:49:10.530090 5177 webserver.cc:460] Webserver started at http://127.27.54.125:38315/ using document root <none> and password file <none>
I20260807 06:49:10.530632 5177 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:10.530797 5177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:10.534368 5177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 06:49:10.536751 5358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:10.537216 5177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 06:49:10.537420 5177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
uuid: "02ef9d212a134bf88539c91ff2ea6089"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "c74bbbed0907f5453e013341adb5f06c"
server_key_iv: "bc4a14c7eb164d9709909160ad5bebb9"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.537679 5177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:10.564420 5177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:10.569329 5177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:10.595748 5177 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.125:44405
I20260807 06:49:10.595816 5432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.125:44405 every 8 connection(s)
I20260807 06:49:10.597014 5177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
I20260807 06:49:10.602502 5434 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:49:10.607437 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Bootstrap starting.
I20260807 06:49:10.628976 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:49:10.629696 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Bootstrap complete.
I20260807 06:49:10.631002 5434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } }
I20260807 06:49:10.631316 5434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02ef9d212a134bf88539c91ff2ea6089, State: Initialized, Role: LEARNER
I20260807 06:49:10.631664 5434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } }
I20260807 06:49:10.632771 5437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } }
I20260807 06:49:10.633078 5434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:10.633180 5437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: LEARNER
I20260807 06:49:10.651312 5452 catalog_manager.cc:1259] Loaded cluster ID: b4ece8fc898c4f84b0e66249daa1d237
I20260807 06:49:10.651582 5452 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: loading cluster ID for follower catalog manager: success
I20260807 06:49:10.654351 5452 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: acquiring CA information for follower catalog manager: success
I20260807 06:49:10.656562 5452 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:49:10.671144 5170 heartbeater.cc:503] Master 127.27.54.126:42485 was elected leader, sending a full tablet report...
I20260807 06:49:10.877013 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.869692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.27.54.125:44405)
0807 06:49:10.870592 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:10.870596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:10.870619 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:49:10.870739 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:10.870743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:10.870827 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:10.871009 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:10.871501 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.872638 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.874314 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.875345 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.875517 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.875828 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:10.876185 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 06:49:10.876369 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":594,"threads_started":1}
I20260807 06:49:10.878178 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:10.869481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43648)
0807 06:49:10.869814 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:10.869839 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:10.870004 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:10.871065 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:10.871072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:10.871103 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:10.871327 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.871336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.872813 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.872819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.874119 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:10.874133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:10.875576 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:10.875582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:10.875836 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:10.875919 (+ 83us) client_negotiation.cc:769] Sending connection context
0807 06:49:10.876044 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 06:49:10.877850 (+ 1806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260807 06:49:10.879670 5399 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 06:49:10.880371 5437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d3f22699b24f46b6bf3d716f39b696. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } }
I20260807 06:49:10.880707 5437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: LEARNER
W20260807 06:49:10.915300 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:10.915691 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:10.915870 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:10.924269 5330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:10.924394 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:10.924474 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:10.924585 5330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:10.924690 5330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:10.935441 5330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.124.keytab
--principal=oryx/127.27.54.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.54.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:10.936995 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:10.938413 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:10.951428 5465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:10.951447 5464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:10.951368 5470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:10.952381 5330 server_base.cc:1034] running on GCE node
I20260807 06:49:10.953250 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:10.954919 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:10.956158 5330 hybrid_clock.cc:648] HybridClock initialized: now 1786085350956123 us; error 58 us; skew 500 ppm
I20260807 06:49:10.956529 5330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:10 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:10.960007 5330 init.cc:377] Logged in from keytab as oryx/127.27.54.124@KRBTEST.COM (short username oryx)
I20260807 06:49:10.961673 5330 webserver.cc:460] Webserver started at http://127.27.54.124:38075/ using document root <none> and password file <none>
I20260807 06:49:10.962435 5330 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:10.962594 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:10.962950 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:10.966197 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/instance:
uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "0d482bd0c0f047226f204f9a4922b3c3"
server_key_iv: "7f98a14dc424f5fc27f844a12ca190b8"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.967141 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal/instance:
uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "0d482bd0c0f047226f204f9a4922b3c3"
server_key_iv: "7f98a14dc424f5fc27f844a12ca190b8"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.973353 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 06:49:10.977555 5485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:10.979214 5330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:10.979451 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "0d482bd0c0f047226f204f9a4922b3c3"
server_key_iv: "7f98a14dc424f5fc27f844a12ca190b8"
server_key_version: "encryptionkey@0"
I20260807 06:49:10.979691 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:11.012218 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:11.016947 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:11.017294 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:11.040138 5330 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.124:38271
I20260807 06:49:11.040179 5555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.124:38271 every 8 connection(s)
I20260807 06:49:11.041710 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
I20260807 06:49:11.045814 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:11.051556 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 5330
I20260807 06:49:11.053689 5556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.124@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.124@KRBTEST.COM
I20260807 06:49:11.074357 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.057278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.27.54.126:42485)
0807 06:49:11.058773 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.058778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.058795 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:49:11.058854 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.058858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.058919 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.059058 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.060447 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.061177 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.062879 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.063290 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.063440 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.067463 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.067469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.067472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.067509 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.070855 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.071740 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.071748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.071752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.071896 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.072401 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.072406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.072408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.072710 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.072885 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.073666 (+ 781us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.073936 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":1217,"threads_started":1}
I20260807 06:49:11.075728 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.054017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.27.54.124:38271)
0807 06:49:11.054916 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.054933 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.054987 (+ 54us) server_negotiation.cc:407] Connection header received
0807 06:49:11.057581 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.057616 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.058166 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.058430 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.060215 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.061466 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.062838 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.063384 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.063612 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.068857 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.068866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.068880 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.068913 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.072003 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.072885 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.072893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.072895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.073034 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.073473 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.073482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.073484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.073854 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.074201 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.074724 (+ 523us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.075019 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":644,"thread_start_us":349,"threads_started":1}
I20260807 06:49:11.075774 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.057008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42344)
0807 06:49:11.057799 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.057851 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.058114 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.059150 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.059157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.059826 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.060230 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.060245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.061345 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.061352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.062643 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.062727 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.063542 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.063549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.063802 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.064568 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.064592 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.067217 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.071021 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.071034 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.071048 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.071581 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.072023 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.072028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.072032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.072246 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.072850 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.072858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.073247 (+ 389us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.073585 (+ 338us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.074149 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":60,"thread_start_us":351,"threads_started":1}
I20260807 06:49:11.078934 5556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.124@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
I20260807 06:49:11.094144 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.079945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43650)
0807 06:49:11.080147 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.080170 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.080287 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.080642 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.080646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.081095 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.081254 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.081263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.082404 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.082408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.083454 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.083468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.084247 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.084251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.084490 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.085243 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.085262 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.087664 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.091358 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.091363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.091366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.091870 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.092269 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.092273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.092275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.092480 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.093076 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.093080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.093249 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.093404 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.093757 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 06:49:11.094153 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.080006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.27.54.125:44405)
0807 06:49:11.080194 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.080203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.080224 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:49:11.080430 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.080434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.080503 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.080589 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.081394 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.082277 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.083612 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.084063 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.084228 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.087924 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.087930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.087933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.087972 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.091189 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.091991 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.091996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.091998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.092146 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.092617 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.092624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.092628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.092945 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.093119 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.093537 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.093795 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:49:11.095753 5556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085350, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.124@KRBTEST.COM for oryx/127.27.54.124@KRBTEST.COM
I20260807 06:49:11.109835 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.096672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.27.54.124:38271)
0807 06:49:11.096902 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.096909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.097003 (+ 94us) server_negotiation.cc:407] Connection header received
0807 06:49:11.097191 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.097196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.097261 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.097361 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.098079 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.098939 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.100066 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.100489 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.100630 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.104317 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.104322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.104326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.104356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.107133 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.107820 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.107825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.107830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.107951 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.108348 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.108356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.108359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.108651 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.108809 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.109167 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.109516 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":195,"server-negotiator.queue_time_us":97}
I20260807 06:49:11.109881 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.096701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.0.0.1:60576)
0807 06:49:11.096901 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.096923 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.097070 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.097379 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.097383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.097795 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.097948 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.097956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.099061 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.099069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.099923 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.099934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.100640 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.100645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.100943 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.101560 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.101579 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.104092 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.107286 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.107291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.107295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.107684 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.108065 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.108071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.108074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.108231 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.108769 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.108774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.108917 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.109092 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.109385 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":25}
I20260807 06:49:11.115629 5556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Bootstrap starting.
I20260807 06:49:11.119388 5556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:11.120687 5556 log.cc:824] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:11.124102 5556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: No bootstrap required, opened a new log
I20260807 06:49:11.132491 5556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } }
I20260807 06:49:11.132880 5556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:11.133054 5556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e85fbd644cc40f1a0d7cfd0147970f7, State: Initialized, Role: FOLLOWER
I20260807 06:49:11.133723 5556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } }
I20260807 06:49:11.134873 5568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } } }
I20260807 06:49:11.135265 5568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:11.136016 5556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:49:11.146242 5585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:49:11.146437 5585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:49:11.150982 5330 master_runner.cc:428] Detected that this master 7e85fbd644cc40f1a0d7cfd0147970f7 is joining an existing cluster
I20260807 06:49:11.151145 5330 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.54.124:38271
I20260807 06:49:11.182961 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.156589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.27.54.124:38271)
0807 06:49:11.156739 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.156743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.165267 (+ 8524us) server_negotiation.cc:407] Connection header received
0807 06:49:11.165735 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.165744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.165842 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.165963 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.167757 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.168852 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.171061 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.171867 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.171953 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.175457 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.175463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.175466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.175496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.179499 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.180361 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.180367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.180371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.180520 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.180976 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.180981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.180984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.181453 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.181592 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.182226 (+ 634us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.182472 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":36}
I20260807 06:49:11.183044 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 4569
I20260807 06:49:11.184180 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.157391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.0.0.1:60580)
0807 06:49:11.165125 (+ 7734us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.165142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.165575 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.166202 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.166208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.167209 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.167596 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.167608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.169104 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.169109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.170924 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.170937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.172324 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.172332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.172724 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.173827 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.173856 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.175283 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.179678 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.179685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.179692 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.180225 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.180644 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.180661 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.180667 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.180847 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.181659 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.181663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.181889 (+ 226us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.183287 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.183585 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7591,"mutex_wait_us":268,"thread_start_us":7613,"threads_started":1}
I20260807 06:49:11.185210 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.156471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.27.54.125:44405)
0807 06:49:11.156669 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.156677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.164426 (+ 7749us) server_negotiation.cc:407] Connection header received
0807 06:49:11.164634 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.164642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.164759 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.164877 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.167389 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.168646 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.171070 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.171788 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.171913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.175428 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.175435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.175439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.175484 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.180430 (+ 4946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.181480 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.181489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.181496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.181664 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.182292 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.182297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.182300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.182629 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.182797 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.183723 (+ 926us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.184766 (+ 1043us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:49:11.186546 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.157374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43652)
0807 06:49:11.164290 (+ 6916us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.164305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.164477 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.165099 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.165105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.166539 (+ 1434us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.167162 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.167178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.168859 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.168863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.170871 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.170887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.172279 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.172292 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.172617 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.173548 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.173587 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.175169 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.180592 (+ 5423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.180599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.180604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.181320 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.181811 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.181821 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.181827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.182099 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.182877 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.182884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.183343 (+ 459us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.185610 (+ 2267us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.185983 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6752,"mutex_wait_us":128,"spinlock_wait_cycles":2944,"thread_start_us":6463,"threads_started":1}
I20260807 06:49:11.189941 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.156390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.27.54.126:42485)
0807 06:49:11.156580 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.156586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.163685 (+ 7099us) server_negotiation.cc:407] Connection header received
0807 06:49:11.164737 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.164744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.164865 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.164946 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.167225 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.168490 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.171041 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.171753 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.171866 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.175261 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.175269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.175274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.175329 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.180211 (+ 4882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.187298 (+ 7087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.187305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.187308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.187538 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.188092 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.188098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.188102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.188518 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.188672 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.189314 (+ 642us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.189586 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 06:49:11.189944 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.156463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42356)
0807 06:49:11.163520 (+ 7057us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.163545 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.164352 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.165356 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.165360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.166481 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.167014 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.167028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.168718 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.168723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.170847 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.170864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.172205 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.172212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.172596 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.173360 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.173393 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.175003 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.180446 (+ 5443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.180451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.180455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.187118 (+ 6663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.187707 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.187712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.187716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.187936 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.188766 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.188771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.188982 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.189149 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.189483 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6872,"mutex_wait_us":189,"thread_start_us":6678,"threads_started":1}
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.65@KRBTEST.COM
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.126@KRBTEST.COM for oryx/127.27.54.124@KRBTEST.COM
I20260807 06:49:11.208424 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.184659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.27.54.65:35161)
0807 06:49:11.185547 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.185557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.185590 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:49:11.185667 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.185675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.185990 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.188391 (+ 2401us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.188621 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.190599 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.193195 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.194064 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.194221 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.200136 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.200159 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.200181 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.200242 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.203891 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.204884 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.204919 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.204923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.205166 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.205683 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.205688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.205690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.206184 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.206866 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.207278 (+ 412us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.207920 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":2192,"threads_started":1}
I20260807 06:49:11.210335 4756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:11.212769 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.195961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.0.0.1:60594)
0807 06:49:11.196234 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.196263 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.196400 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.196833 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.196839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.197262 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.197446 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.197456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.198562 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.198568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.200013 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.200031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.201473 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.201479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.201690 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.202345 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.202371 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.205302 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.209576 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.209582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.209586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.209982 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.210397 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.210401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.210405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.210573 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.211443 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.211451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.211699 (+ 248us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.211865 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.212258 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260807 06:49:11.213900 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.195978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.27.54.124:38271)
0807 06:49:11.196144 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.196150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.196343 (+ 193us) server_negotiation.cc:407] Connection header received
0807 06:49:11.196568 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.196580 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.196671 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.196793 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.197614 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.198399 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.200196 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.200953 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.201096 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.205492 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.205497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.205501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.205534 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.209405 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.210118 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.210124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.210128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.210263 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.210763 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.210768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.210774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.211187 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.211368 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.213157 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.213453 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:49:11.215638 4480 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.54.124:38271: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
member_type: NON_VOTER
last_known_addr {
host: "127.27.54.124"
port: 38271
}
attrs {
promote: true
}
}
dest_uuid: "d8d3f22699b24f46b6bf3d716f39b696"
cas_config_opid_index: 5
I20260807 06:49:11.216830 4480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } }
I20260807 06:49:11.218842 5321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.219616 5399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 LEARNER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 06:49:11.223831 5516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:11.227063 5321 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.54.124:38271
I20260807 06:49:11.227334 5600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 06:49:11.240432 5600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.241066 5600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:11.241552 5399 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.246824 5437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: NON_VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: true } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.247136 5437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: LEARNER
I20260807 06:49:11.248451 5321 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: attempting to promote NON_VOTER 02ef9d212a134bf88539c91ff2ea6089 to VOTER
I20260807 06:49:11.249402 5321 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } }
I20260807 06:49:11.253666 5398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 LEARNER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 06:49:11.254472 5599 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 06:49:11.257253 5599 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.261689 5399 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.262455 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.262838 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:11.263155 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.263376 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:11.263327 5516 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 06:49:11.263906 5515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 06:49:11.264515 5330 master.cc:537] Master@127.27.54.124:38271 shutting down...
I20260807 06:49:11.264916 5568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:11.265369 5568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } }
I20260807 06:49:11.266090 5437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.266060 5605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:11.266378 5437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:11.268139 5568 leader_election.cc:288] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125:44405)
I20260807 06:49:11.269428 4492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d3f22699b24f46b6bf3d716f39b696" is_pre_election: true
W20260807 06:49:11.269834 5491 tablet.cc:2369] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:49:11.281216 5330 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:11.281648 5330 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:11.281910 5330 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: stopping tablet replica
W20260807 06:49:11.283111 4453 consensus_peers.cc:487] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 -> Peer 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124:38271): Couldn't send request to peer 7e85fbd644cc40f1a0d7cfd0147970f7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 06:49:11.284655 5399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ef9d212a134bf88539c91ff2ea6089" is_pre_election: true
I20260807 06:49:11.285560 5487 leader_election.cc:302] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e85fbd644cc40f1a0d7cfd0147970f7; no voters: 02ef9d212a134bf88539c91ff2ea6089, d8d3f22699b24f46b6bf3d716f39b696
W20260807 06:49:11.285933 5487 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260807 06:49:11.290045 5330 master.cc:552] Master@127.27.54.124:38271 shutdown complete.
I20260807 06:49:11.290174 5330 master_runner.cc:305] Clearing existing system tablet
I20260807 06:49:11.290607 5330 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:11.290791 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:11.295308 5330 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:11.299139 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:11.299611 5330 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:11.300040 5330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260807 06:49:11.300207 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "0d482bd0c0f047226f204f9a4922b3c3"
server_key_iv: "7f98a14dc424f5fc27f844a12ca190b8"
server_key_version: "encryptionkey@0"
I20260807 06:49:11.305727 5330 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:11.313498 5330 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 06:49:11.313792 5330 log.cc:1196] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 06:49:11.314410 5330 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Deleting consensus metadata
I20260807 06:49:11.314765 5330 master_runner.cc:315] Copying system tablet from 127.27.54.126:42485
I20260807 06:49:11.317457 5330 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: tablet copy: Beginning tablet copy session from remote peer at address 127.27.54.126:42485
I20260807 06:49:11.343582 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.318386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42362)
0807 06:49:11.327224 (+ 8838us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.327264 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.327489 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.327972 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.327976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.328605 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.328970 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.328986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.330573 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.330579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.332234 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.332251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.333378 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.333384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.333584 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.334433 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.334457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.335726 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.340300 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.340305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.340308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.340880 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.341358 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.341364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.341368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.341545 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.342310 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.342315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.342503 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.342693 (+ 190us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.343051 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8666,"thread_start_us":8505,"threads_started":1}
I20260807 06:49:11.344695 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.318446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.27.54.126:42485)
0807 06:49:11.318722 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.318729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.327435 (+ 8706us) server_negotiation.cc:407] Connection header received
0807 06:49:11.327645 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.327651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.327780 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.327903 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.329181 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.330396 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.332421 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.333177 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.333321 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.335987 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.335994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.336000 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.336045 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.340118 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.341043 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.341049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.341053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.341218 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.341694 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.341699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.341703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.342073 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.342240 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.344120 (+ 1880us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.344351 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260807 06:49:11.346230 4507 tablet_copy_service.cc:140] P d8d3f22699b24f46b6bf3d716f39b696: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7e85fbd644cc40f1a0d7cfd0147970f7 ({username='oryx', principal='oryx/127.27.54.124@KRBTEST.COM'} at 127.0.0.1:42362)
I20260807 06:49:11.346418 4507 tablet_copy_service.cc:161] P d8d3f22699b24f46b6bf3d716f39b696: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7e85fbd644cc40f1a0d7cfd0147970f7 at {username='oryx', principal='oryx/127.27.54.124@KRBTEST.COM'} at 127.0.0.1:42362: session id = 7e85fbd644cc40f1a0d7cfd0147970f7-00000000000000000000000000000000
I20260807 06:49:11.350008 4507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 06:49:11.352046 5330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 06:49:11.356825 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 4569
I20260807 06:49:11.359870 5330 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: tablet copy: Starting download of 0 data blocks...
I20260807 06:49:11.360203 5330 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: tablet copy: Starting download of 1 WAL segments...
I20260807 06:49:11.364791 5330 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 06:49:11.368525 4507 tablet_copy_service.cc:342] P d8d3f22699b24f46b6bf3d716f39b696: Request end of tablet copy session 7e85fbd644cc40f1a0d7cfd0147970f7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.54.124@KRBTEST.COM'} at 127.0.0.1:42362
I20260807 06:49:11.368682 4507 tablet_copy_service.cc:433] P d8d3f22699b24f46b6bf3d716f39b696: ending tablet copy session 7e85fbd644cc40f1a0d7cfd0147970f7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7e85fbd644cc40f1a0d7cfd0147970f7
I20260807 06:49:11.380887 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:11.384501 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:11.389612 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:11.392334 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:11.392627 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:35161
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:42485
--webserver_port=36719
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:11.393271 5330 server_base.cc:1034] running on GCE node
I20260807 06:49:11.393704 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:11.394508 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:11.395928 5330 hybrid_clock.cc:648] HybridClock initialized: now 1786085351395845 us; error 68 us; skew 500 ppm
I20260807 06:49:11.396252 5330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085351, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:11.398725 5330 init.cc:377] Logged in from keytab as oryx/127.27.54.124@KRBTEST.COM (short username oryx)
I20260807 06:49:11.403399 5330 webserver.cc:460] Webserver started at http://127.27.54.124:45599/ using document root <none> and password file <none>
I20260807 06:49:11.403781 5330 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:11.403900 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:11.407224 5330 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 06:49:11.409291 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:11.409796 5330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 06:49:11.409974 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
format_stamp: "Formatted at 2026-08-07 06:49:10 on dist-test-slave-3kk6"
server_key: "0d482bd0c0f047226f204f9a4922b3c3"
server_key_iv: "7f98a14dc424f5fc27f844a12ca190b8"
server_key_version: "encryptionkey@0"
I20260807 06:49:11.410239 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:11.429229 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:11.433785 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:11.454383 5330 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.124:38271
I20260807 06:49:11.454414 5715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.124:38271 every 8 connection(s)
I20260807 06:49:11.455719 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
I20260807 06:49:11.461187 5717 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:49:11.465337 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Bootstrap starting.
I20260807 06:49:11.484304 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:49:11.484894 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: Bootstrap complete.
I20260807 06:49:11.486063 5717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } }
I20260807 06:49:11.486379 5717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e85fbd644cc40f1a0d7cfd0147970f7, State: Initialized, Role: LEARNER
I20260807 06:49:11.486713 5717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } }
I20260807 06:49:11.487995 5717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:11.487807 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.488353 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: This master's current role is: LEARNER
I20260807 06:49:11.499022 5737 catalog_manager.cc:1259] Loaded cluster ID: b4ece8fc898c4f84b0e66249daa1d237
I20260807 06:49:11.499132 5737 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: loading cluster ID for follower catalog manager: success
I20260807 06:49:11.502787 5737 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: acquiring CA information for follower catalog manager: success
I20260807 06:49:11.505345 5737 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 06:49:11.789893 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:11.790195 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:11.790395 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:11.798835 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:11.798960 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:11.799183 5631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:11.810325 5631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:35161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=36719
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:11.811933 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:11.813350 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:11.828245 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:11.828276 5749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:11.828233 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:11.832125 5631 server_base.cc:1034] running on GCE node
I20260807 06:49:11.832844 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:11.833963 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:11.835209 5631 hybrid_clock.cc:648] HybridClock initialized: now 1786085351835153 us; error 39 us; skew 500 ppm
I20260807 06:49:11.835624 5631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085351, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:11.843075 5631 init.cc:377] Logged in from keytab as oryx/127.27.54.65@KRBTEST.COM (short username oryx)
I20260807 06:49:11.845067 5631 webserver.cc:460] Webserver started at http://127.27.54.65:36719/ using document root <none> and password file <none>
I20260807 06:49:11.846029 5631 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:11.846195 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:11.852072 5631 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 06:49:11.855026 5758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:11.856431 5631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:11.856638 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "12c49bf3489f4112ab6bde0f7612c3b9"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "072bd4b573bccf5e5853cab1f75ee15a"
server_key_iv: "a8fe649914a8311b9afe660d21ad2d0a"
server_key_version: "encryptionkey@0"
I20260807 06:49:11.857674 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:11.887014 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:11.893050 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:11.893404 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:11.894883 5631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:11.896903 5631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:11.897042 5631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:11.897189 5631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:11.897317 5631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:11.899461 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.891371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.0.0.1:60606)
0807 06:49:11.892266 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.892295 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.892477 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.893514 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.893520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.893554 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:11.893788 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.893801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.895896 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.895901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.897218 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.897233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.898495 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.898501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.898729 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.898772 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.898901 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.899060 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":510,"threads_started":1}
I20260807 06:49:11.899456 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.891540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.27.54.124:38271)
0807 06:49:11.893103 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.893108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.893125 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:49:11.893246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.893253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.893340 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.893443 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:11.894017 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.895700 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.897414 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.898275 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.898443 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.898716 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.898931 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.899070 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1450,"thread_start_us":1316,"threads_started":1}
I20260807 06:49:11.901012 5678 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260807 06:49:11.901736 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d3f22699b24f46b6bf3d716f39b696. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:11.902097 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: This master's current role is: LEARNER
I20260807 06:49:11.939910 5631 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:35161
I20260807 06:49:11.940037 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:35161 every 8 connection(s)
I20260807 06:49:11.941468 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:11.947736 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 5631
I20260807 06:49:11.948192 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 4771
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085351, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.124@KRBTEST.COM
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085351, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085351, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
Aug 07 06:49:11 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.66@KRBTEST.COM
I20260807 06:49:11.990820 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.945191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:60681 (local address 127.27.54.124:38271)
0807 06:49:11.945426 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.945433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.960690 (+ 15257us) server_negotiation.cc:407] Connection header received
0807 06:49:11.965776 (+ 5086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.965786 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.965905 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.965993 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.969043 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.970536 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.973676 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.974437 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.974552 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.981047 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.981055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.981062 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.981139 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.985484 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.987180 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.987187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.987191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.987364 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.988145 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.988151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.988154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.988637 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.988824 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.990054 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.990352 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 06:49:11.993628 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.945894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.27.54.65:60681)
0807 06:49:11.960398 (+ 14504us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.960482 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.965325 (+ 4843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.966713 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.966728 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.967739 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.968760 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.968792 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.970822 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.970831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.973467 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.973488 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.975071 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.975081 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.975493 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.976555 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.976601 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.980747 (+ 4146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.985657 (+ 4910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.985669 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.985696 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.986477 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.987557 (+ 1080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.987562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.987566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.988019 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.988985 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.988995 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.989582 (+ 587us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.991130 (+ 1548us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.991809 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14016,"thread_start_us":14744,"threads_started":1}
I20260807 06:49:11.995498 5929 heartbeater.cc:347] Connected to a master server at 127.27.54.124:38271
I20260807 06:49:11.995735 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.945172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:39755 (local address 127.27.54.126:42485)
0807 06:49:11.945892 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.945898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.975053 (+ 29155us) server_negotiation.cc:407] Connection header received
0807 06:49:11.975250 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.975254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.975324 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.975461 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.977086 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.978322 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.980629 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.981356 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.981459 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.987702 (+ 6243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.987709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.987715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.987759 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.991860 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.993028 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.993035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.993041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.993213 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.993683 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.993690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.993694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.994323 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.994470 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.995059 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.995319 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":409,"threads_started":1}
I20260807 06:49:11.995837 5929 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:11.996542 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.946289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.27.54.65:51963)
0807 06:49:11.973647 (+ 27358us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.973681 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.973889 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.977913 (+ 4024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.977918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.978643 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.979241 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.979257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.980996 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.981002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.982884 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.982902 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.984153 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.984161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.984370 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.984982 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.985009 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.987836 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.992758 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.992763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.992766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.993337 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.994015 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.994021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.994024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.994210 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.995242 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.995247 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.995414 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.995554 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.995913 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27162,"mutex_wait_us":1,"thread_start_us":28221,"threads_started":1}
I20260807 06:49:11.996913 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.946578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:39755)
0807 06:49:11.974902 (+ 28324us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:11.974917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:11.975100 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:11.975699 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:11.975704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:11.976414 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:11.976852 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.976866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.978591 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.978598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.980432 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:11.980451 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.981849 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.981858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:11.982187 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.982869 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:11.982893 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:11.987365 (+ 4472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:11.992088 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.992094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.992099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.992864 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.993377 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:11.993381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:11.993384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:11.993531 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:11.994558 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:11.994563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:11.994768 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 06:49:11.996001 (+ 1233us) client_negotiation.cc:240] Negotiation successful
0807 06:49:11.996335 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28198,"mutex_wait_us":87,"thread_start_us":27416,"threads_started":1}
I20260807 06:49:11.996989 5929 heartbeater.cc:511] Master 127.27.54.124:38271 requested a full tablet report, sending...
I20260807 06:49:11.997902 5927 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42485
I20260807 06:49:11.998051 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.946223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51963 (local address 127.27.54.125:44405)
0807 06:49:11.977377 (+ 31154us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.977385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.977412 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:49:11.977476 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.977481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.977587 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.979423 (+ 1836us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.979619 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.980820 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.983100 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.983872 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.984702 (+ 830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.988118 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.988126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.988130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.988169 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.992577 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.993510 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.993515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.993518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.993701 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.994372 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.994376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.994379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.995098 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.997251 (+ 2153us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.997394 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.997647 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31004,"thread_start_us":31494,"threads_started":1}
I20260807 06:49:11.998072 5927 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:11.999648 5658 ts_manager.cc:194] Registered new tserver with Master: 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:12.000304 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:11.949635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.27.54.66:36113)
0807 06:49:11.971687 (+ 22052us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:11.971697 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:11.972565 (+ 868us) server_negotiation.cc:407] Connection header received
0807 06:49:11.972814 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:11.972824 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:11.973100 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:11.973279 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:11.976323 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.978185 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.980846 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:11.981930 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:11.982099 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:11.989743 (+ 7644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:11.989766 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:11.989787 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:11.989832 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:11.994589 (+ 4757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.996103 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.996112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.996115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.996286 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:11.996908 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:11.996914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:11.996917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:11.997482 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:11.997827 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:11.999505 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0807 06:49:11.999828 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21796,"thread_start_us":22044,"threads_started":1}
I20260807 06:49:12.001150 5930 heartbeater.cc:347] Connected to a master server at 127.27.54.125:44405
I20260807 06:49:12.001308 5930 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:12.001789 4962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:12.001863 5930 heartbeater.cc:511] Master 127.27.54.125:44405 requested a full tablet report, sending...
I20260807 06:49:12.002485 5927 heartbeater.cc:511] Master 127.27.54.126:42485 requested a full tablet report, sending...
I20260807 06:49:12.003976 4480 ts_manager.cc:194] Re-registered known tserver with Master: 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:12.003924 5384 ts_manager.cc:194] Registered new tserver with Master: 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:12.005275 4480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:39755
I20260807 06:49:12.088091 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 4771
I20260807 06:49:12.117839 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:36113
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:42485
--webserver_port=45763
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:12.486840 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:12.487202 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:12.487382 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 06:49:12.491410 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: NON_VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: true } } }
I20260807 06:49:12.491765 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:12.494148 5951 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: attempting to promote NON_VOTER 7e85fbd644cc40f1a0d7cfd0147970f7 to VOTER
I20260807 06:49:12.494876 5951 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:12.497367 5678 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 LEARNER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 06:49:12.497632 5399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Refusing update from remote peer d8d3f22699b24f46b6bf3d716f39b696: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20260807 06:49:12.497844 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:12.497998 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260807 06:49:12.498003 5951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260807 06:49:12.498227 5944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:12.498449 5605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 06:49:12.502120 5605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.502935 5399 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.503521 5678 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.510399 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.510501 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.510797 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:12.510874 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:12.511066 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.511315 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:12.512193 5964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.511965 5965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d3f22699b24f46b6bf3d716f39b696" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:12.512490 5964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:12.512560 5965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:12.514573 5944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:36113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=45763
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:12.516125 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:12.517558 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:12.532011 5969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:12.532034 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:12.532785 5970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:12.532994 5944 server_base.cc:1034] running on GCE node
I20260807 06:49:12.533732 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:12.534853 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:12.536109 5944 hybrid_clock.cc:648] HybridClock initialized: now 1786085352536027 us; error 59 us; skew 500 ppm
I20260807 06:49:12.536502 5944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:12 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085352, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:12.540400 5944 init.cc:377] Logged in from keytab as oryx/127.27.54.66@KRBTEST.COM (short username oryx)
I20260807 06:49:12.542030 5944 webserver.cc:460] Webserver started at http://127.27.54.66:45763/ using document root <none> and password file <none>
I20260807 06:49:12.542852 5944 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:12.542991 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:12.548743 5944 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:49:12.551636 5984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:12.553131 5944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:49:12.553293 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "5ec855a623ec4065b270632de521ca74"
format_stamp: "Formatted at 2026-08-07 06:49:08 on dist-test-slave-3kk6"
server_key: "0e13d4f24d3c0f04fd417fc63f42458a"
server_key_iv: "201e3d1cacd123077b250e0dfb22ca1f"
server_key_version: "encryptionkey@0"
I20260807 06:49:12.554343 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:12.570308 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:12.574764 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:12.575111 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:12.576195 5944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:12.577867 5944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:12.577980 5944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:12.578100 5944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:12.578187 5944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:12.616691 5944 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:36113
I20260807 06:49:12.616753 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:36113 every 8 connection(s)
I20260807 06:49:12.618360 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:12.628731 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 5944
I20260807 06:49:12.629046 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 4980
Aug 07 06:49:12 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085352, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.124@KRBTEST.COM
Aug 07 06:49:12 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085352, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
Aug 07 06:49:12 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085352, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
I20260807 06:49:12.659765 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.623136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:51913 (local address 127.27.54.124:38271)
0807 06:49:12.623705 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:12.623712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:12.634027 (+ 10315us) server_negotiation.cc:407] Connection header received
0807 06:49:12.634364 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:12.634374 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:12.634496 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:12.634597 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:12.636740 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.638318 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.641784 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.642769 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.642913 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.649307 (+ 6394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:12.649317 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:12.649323 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:12.649376 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:12.653898 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.655697 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.655704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.655707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.655899 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.656536 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.656545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.656549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.656927 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:12.657026 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:12.658743 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0807 06:49:12.659198 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":864,"threads_started":1}
I20260807 06:49:12.660451 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.622683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:39015 (local address 127.27.54.126:42485)
0807 06:49:12.623309 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:12.623318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:12.626714 (+ 3396us) server_negotiation.cc:407] Connection header received
0807 06:49:12.630417 (+ 3703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:12.630427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:12.630552 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:12.630709 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:12.634325 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.635893 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.646837 (+ 10944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.647599 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.647709 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.652456 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:12.652465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:12.652470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:12.652514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:12.656619 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.657542 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.657548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.657551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.657754 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.658323 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.658329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.658332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.658741 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:12.658921 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:12.659544 (+ 623us) server_negotiation.cc:299] Negotiation successful
0807 06:49:12.659798 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":658,"threads_started":1}
I20260807 06:49:12.662711 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.622310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:39015)
0807 06:49:12.624651 (+ 2341us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:12.624702 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:12.629773 (+ 5071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:12.631434 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:12.631452 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:12.632737 (+ 1285us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:12.633980 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:12.634012 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.636152 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.636159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:12.645297 (+ 9138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:12.645320 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.648123 (+ 2803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.648133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:12.648414 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.649152 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:12.649180 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:12.652143 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:12.656796 (+ 4653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:12.656803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:12.656808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:12.657326 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:12.657929 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:12.657934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:12.657937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:12.658160 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:12.659030 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:12.659035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:12.659201 (+ 166us) client_negotiation.cc:769] Sending connection context
0807 06:49:12.660896 (+ 1695us) client_negotiation.cc:240] Negotiation successful
0807 06:49:12.661285 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1998,"thread_start_us":10805,"threads_started":1}
I20260807 06:49:12.662797 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.624168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.27.54.66:51913)
0807 06:49:12.633869 (+ 9701us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:12.633898 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:12.634165 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:12.634989 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:12.634998 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:12.635930 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:12.636426 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:12.636442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.638589 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.638597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:12.641533 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:12.641556 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.643440 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.643451 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:12.643893 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.644930 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:12.644974 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:12.648983 (+ 4009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:12.654080 (+ 5097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:12.654090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:12.654110 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:12.655536 (+ 1426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:12.656069 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:12.656074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:12.656078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:12.656365 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:12.657133 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:12.657140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:12.658214 (+ 1074us) client_negotiation.cc:769] Sending connection context
0807 06:49:12.660050 (+ 1836us) client_negotiation.cc:240] Negotiation successful
0807 06:49:12.660666 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9449,"thread_start_us":3335,"threads_started":1}
Aug 07 06:49:12 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.67@KRBTEST.COM
I20260807 06:49:12.664736 6160 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42485
I20260807 06:49:12.664810 6161 heartbeater.cc:347] Connected to a master server at 127.27.54.124:38271
I20260807 06:49:12.665125 6161 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:12.665134 6160 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:12.666301 6161 heartbeater.cc:511] Master 127.27.54.124:38271 requested a full tablet report, sending...
I20260807 06:49:12.666724 6160 heartbeater.cc:511] Master 127.27.54.126:42485 requested a full tablet report, sending...
I20260807 06:49:12.667303 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.623729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.27.54.66:34081)
0807 06:49:12.627892 (+ 4163us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:12.627919 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:12.630070 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:12.647436 (+ 17366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:12.647442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:12.648088 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:12.648628 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:12.648645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.650673 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.650680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:12.652609 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:12.652625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.653961 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.653971 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:12.654200 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.654850 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:12.654878 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:12.658212 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:12.663663 (+ 5451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:12.663668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:12.663672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:12.664245 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:12.664768 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:12.664775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:12.664780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:12.664990 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:12.665908 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:12.665918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:12.666157 (+ 239us) client_negotiation.cc:769] Sending connection context
0807 06:49:12.666311 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 06:49:12.666673 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4001,"mutex_wait_us":3,"spinlock_wait_cycles":42368,"thread_start_us":648,"threads_started":1}
I20260807 06:49:12.668450 4480 ts_manager.cc:194] Re-registered known tserver with Master: 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113)
I20260807 06:49:12.668578 5658 ts_manager.cc:194] Registered new tserver with Master: 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113)
I20260807 06:49:12.668684 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.621784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:34081 (local address 127.27.54.125:44405)
0807 06:49:12.646987 (+ 25203us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:12.646996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:12.647020 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:49:12.647090 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:12.647098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:12.647190 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:12.648798 (+ 1608us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:12.649108 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.650445 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.652829 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.653707 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.654561 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.658494 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:12.658502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:12.658510 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:12.658556 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:12.663481 (+ 4925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.664428 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.664433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.664436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.664613 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.665339 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.665344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.665358 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.665752 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:12.667771 (+ 2019us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:12.667926 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 06:49:12.668167 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25018,"thread_start_us":22938,"threads_started":1}
I20260807 06:49:12.669644 6163 heartbeater.cc:347] Connected to a master server at 127.27.54.125:44405
I20260807 06:49:12.669800 6163 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:12.669795 4480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:39015
I20260807 06:49:12.670418 6163 heartbeater.cc:511] Master 127.27.54.125:44405 requested a full tablet report, sending...
I20260807 06:49:12.671561 5384 ts_manager.cc:194] Registered new tserver with Master: 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113)
I20260807 06:49:12.674324 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:12.630277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.27.54.67:44931)
0807 06:49:12.646507 (+ 16230us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:12.646513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:12.646538 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:49:12.646593 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:12.646599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:12.646947 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:12.648068 (+ 1121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:12.651873 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.653752 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.655953 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:12.657810 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:12.657934 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:12.666112 (+ 8178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:12.666135 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:12.666152 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:12.666197 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:12.670997 (+ 4800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.672006 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.672015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.672018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.672255 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:12.672720 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:12.672724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:12.672726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:12.673115 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:12.673377 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:12.673661 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 06:49:12.673879 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16034,"thread_start_us":15872,"threads_started":1}
I20260807 06:49:12.675717 5160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:12.762094 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 4980
I20260807 06:49:12.786707 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:44931
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:42485
--webserver_port=37455
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:13.007836 5927 heartbeater.cc:503] Master 127.27.54.126:42485 was elected leader, sending a full tablet report...
W20260807 06:49:13.146121 6181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:13.146476 6181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:13.146693 6181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:13.154065 6181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:13.154241 6181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:13.154433 6181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:13.164990 6181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:44931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=37455
--tserver_master_addrs=127.27.54.126:42485,127.27.54.125:44405,127.27.54.124:38271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:13.166522 6181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:13.167894 6181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:13.181777 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:13.181876 6188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:13.182331 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:13.183066 6181 server_base.cc:1034] running on GCE node
I20260807 06:49:13.183760 6181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:13.184775 6181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:13.186017 6181 hybrid_clock.cc:648] HybridClock initialized: now 1786085353185976 us; error 28 us; skew 500 ppm
I20260807 06:49:13.186328 6181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:13 dist-test-slave-3kk6 krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:13.190233 6181 init.cc:377] Logged in from keytab as oryx/127.27.54.67@KRBTEST.COM (short username oryx)
I20260807 06:49:13.192009 6181 webserver.cc:460] Webserver started at http://127.27.54.67:37455/ using document root <none> and password file <none>
I20260807 06:49:13.192864 6181 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:13.193038 6181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:13.198968 6181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 06:49:13.201713 6204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:13.203243 6181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:13.203437 6181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "b5b11280d810438b9c5efb9800a5e40a"
format_stamp: "Formatted at 2026-08-07 06:49:09 on dist-test-slave-3kk6"
server_key: "321a5e26bca4d4f1ad903f2b132514c5"
server_key_iv: "11af0ac5b9e0f3bf0914af1c99b14196"
server_key_version: "encryptionkey@0"
I20260807 06:49:13.204602 6181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:13.220044 6181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:13.224906 6181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:13.225245 6181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:13.226325 6181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:13.227842 6181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:13.227988 6181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:13.228130 6181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:13.228226 6181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:13.265631 6181 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:44931
I20260807 06:49:13.265689 6359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:44931 every 8 connection(s)
I20260807 06:49:13.267202 6181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:13.276191 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 6181
Aug 07 06:49:13 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.125@KRBTEST.COM
Aug 07 06:49:13 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
Aug 07 06:49:13 dist-test-slave-3kk6 krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.124@KRBTEST.COM
I20260807 06:49:13.317391 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.270486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:45673 (local address 127.27.54.125:44405)
0807 06:49:13.285922 (+ 15436us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.285932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.289670 (+ 3738us) server_negotiation.cc:407] Connection header received
0807 06:49:13.291305 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.291311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.291406 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.291522 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.293885 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.295020 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.299726 (+ 4706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.300398 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.300524 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.305108 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.305117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.305122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.305167 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.312615 (+ 7448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.313929 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.313936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.313941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.314168 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.314843 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.314848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.314851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.315314 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.315467 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.316647 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.316903 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15230,"thread_start_us":15086,"threads_started":1}
I20260807 06:49:13.319310 6373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.276548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:49503 (local address 127.27.54.126:42485)
0807 06:49:13.290701 (+ 14153us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.290710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.293287 (+ 2577us) server_negotiation.cc:407] Connection header received
0807 06:49:13.293380 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.293386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.293483 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.293643 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.295235 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.296375 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.298633 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.299331 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.299468 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.306409 (+ 6941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.306417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.306422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.306468 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.312575 (+ 6107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.316825 (+ 4250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.316832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.316835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.316999 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.317515 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.317523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.317528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.317918 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.318064 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.318641 (+ 577us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.318915 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13969,"thread_start_us":14153,"threads_started":1}
I20260807 06:49:13.320662 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.276594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:49503)
0807 06:49:13.291691 (+ 15097us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.291713 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.291875 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.293660 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.293665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.294346 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:13.294636 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.294686 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.296508 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.296516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.298432 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.298468 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.299779 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.299786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.300149 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.301216 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:13.301256 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:13.305426 (+ 4170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:13.316089 (+ 10663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:13.316093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:13.316097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:13.316655 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:13.317162 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:13.317167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:13.317171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:13.317356 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:13.318149 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:13.318154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:13.318312 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.319834 (+ 1522us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.320130 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14471,"mutex_wait_us":171,"thread_start_us":10347,"threads_started":1}
I20260807 06:49:13.322747 6361 heartbeater.cc:347] Connected to a master server at 127.27.54.126:42485
I20260807 06:49:13.323134 6361 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:13.324190 6361 heartbeater.cc:511] Master 127.27.54.126:42485 requested a full tablet report, sending...
I20260807 06:49:13.325078 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.270469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:57369 (local address 127.27.54.124:38271)
0807 06:49:13.286375 (+ 15906us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.286385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.287818 (+ 1433us) server_negotiation.cc:407] Connection header received
0807 06:49:13.291328 (+ 3510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.291333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.291414 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.291501 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.294298 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.295927 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.298697 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.299350 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.299468 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.309210 (+ 9742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.309219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.309227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.309276 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.315940 (+ 6664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.321213 (+ 5273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.321220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.321225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.321436 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.322674 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.322681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.322684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.323162 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.323304 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.324380 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.324636 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15726,"thread_start_us":15898,"threads_started":1}
I20260807 06:49:13.326331 4480 ts_manager.cc:194] Re-registered known tserver with Master: b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931)
I20260807 06:49:13.327654 4480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:49503
I20260807 06:49:13.329173 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.271377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.27.54.67:57369)
0807 06:49:13.287439 (+ 16062us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.287500 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.290874 (+ 3374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.291911 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.291928 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.292640 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:13.294053 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.294072 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.296205 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.296213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.298441 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.298469 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.299731 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.299743 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.300159 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.305653 (+ 5494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:13.305677 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:13.308891 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:13.320498 (+ 11607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:13.320503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:13.320507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:13.321046 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:13.322261 (+ 1215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:13.322266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:13.322269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:13.322462 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:13.323440 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:13.323444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:13.323687 (+ 243us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.325344 (+ 1657us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.328700 (+ 3356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15697,"mutex_wait_us":94,"spinlock_wait_cycles":51456,"thread_start_us":19337,"threads_started":1}
I20260807 06:49:13.330796 6363 heartbeater.cc:347] Connected to a master server at 127.27.54.124:38271
I20260807 06:49:13.330960 6363 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:13.331385 6363 heartbeater.cc:511] Master 127.27.54.124:38271 requested a full tablet report, sending...
I20260807 06:49:13.332840 5658 ts_manager.cc:194] Registered new tserver with Master: b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931)
I20260807 06:49:13.320374 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.271377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.27.54.67:45673)
0807 06:49:13.289428 (+ 18051us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.289452 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.290895 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.291919 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.291927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.292858 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:13.293569 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.293608 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.295173 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.295178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.298432 (+ 3254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.298468 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.300847 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.300854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.301060 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.301673 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:13.301696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:13.304797 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:13.312825 (+ 8028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:13.312839 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:13.312864 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:13.313750 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:13.314357 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:13.314364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:13.314369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:13.314687 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:13.315573 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:13.315582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:13.316086 (+ 504us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.318119 (+ 2033us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.318772 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17867,"mutex_wait_us":130,"spinlock_wait_cycles":37760,"thread_start_us":17495,"threads_started":1}
I20260807 06:49:13.335490 6364 heartbeater.cc:347] Connected to a master server at 127.27.54.125:44405
I20260807 06:49:13.335642 6364 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:13.336251 6364 heartbeater.cc:511] Master 127.27.54.125:44405 requested a full tablet report, sending...
I20260807 06:49:13.337868 5384 ts_manager.cc:194] Registered new tserver with Master: b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931)
I20260807 06:49:13.338370 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.300044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.27.54.124:38271)
0807 06:49:13.308160 (+ 8116us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.308167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.308203 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:49:13.308281 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.308289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.308384 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.308509 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.317789 (+ 9280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.319406 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.322131 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.322954 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.323058 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.328353 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.328360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.328364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.328436 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.333734 (+ 5298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.334948 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.334958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.334962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.335143 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.335698 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.335703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.335706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.336107 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.336392 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.337583 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.337863 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7949,"thread_start_us":7808,"threads_started":1}
I20260807 06:49:13.339524 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.307103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.27.54.125:44405)
0807 06:49:13.318051 (+ 10948us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.318056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.318405 (+ 349us) server_negotiation.cc:407] Connection header received
0807 06:49:13.319645 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.319649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.319739 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.319814 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.322861 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.323984 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.326303 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.327105 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.327208 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.331406 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.331413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.331416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.331452 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.335501 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.336464 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.336470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.336473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.336692 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.337217 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.337222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.337226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.337702 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.338033 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.339092 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.339273 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10804,"thread_start_us":12355,"threads_started":1}
I20260807 06:49:13.347319 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.299735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.27.54.126:42485)
0807 06:49:13.321846 (+ 22111us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.321852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.321869 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:49:13.321913 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.321917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.321993 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.328387 (+ 6394us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.332636 (+ 4249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.333940 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.336428 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.337218 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.337354 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.341449 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.341455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.341458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.341492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.344291 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.344995 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.345001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.345005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.345156 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.345623 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.345629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.345632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.345949 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.346216 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.346611 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.346894 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10782,"thread_start_us":9483,"threads_started":1}
I20260807 06:49:13.351676 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:49:13.360018 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:13.429580 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.393181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:44931 (local address 127.0.0.1:47644)
0807 06:49:13.419095 (+ 25914us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.419115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.419322 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.420109 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.420116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.420145 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:13.420690 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.420707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.423505 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.423510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.426050 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.426065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.428611 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.428618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.428877 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.428917 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.429022 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.429159 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25784,"thread_start_us":24019,"threads_started":1}
I20260807 06:49:13.429853 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.393995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.27.54.67:44931)
0807 06:49:13.418292 (+ 24297us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.418305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.419291 (+ 986us) server_negotiation.cc:407] Connection header received
0807 06:49:13.419496 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.419505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.419814 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.420096 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:13.420975 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.423318 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.426291 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.428367 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.428524 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.428817 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.429089 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.429327 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24057,"thread_start_us":24319,"threads_started":1}
I20260807 06:49:13.434978 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.393583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:35161 (local address 127.0.0.1:37612)
0807 06:49:13.419989 (+ 26406us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.420003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.420171 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.421186 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.421189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.421208 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:13.421678 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.421686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.426701 (+ 5015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.426706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.431682 (+ 4976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.431696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.434065 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.434071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.434247 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.434278 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.434388 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.434521 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26310,"thread_start_us":33621,"threads_started":1}
I20260807 06:49:13.435189 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.394262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.27.54.65:35161)
0807 06:49:13.416601 (+ 22339us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.416616 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.420344 (+ 3728us) server_negotiation.cc:407] Connection header received
0807 06:49:13.420493 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.420504 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.420740 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.420957 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:13.421898 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.426517 (+ 4619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.431900 (+ 5383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.433510 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.433641 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.433835 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.434505 (+ 670us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.434761 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21994,"thread_start_us":22372,"threads_started":1}
I20260807 06:49:13.435690 6270 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6559622f3e402a8b215fb266d10472 (DEFAULT_TABLE table=test-table [id=0e47ad2eb4a144e291e471a986a5cb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:13.437136 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6559622f3e402a8b215fb266d10472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:13.443825 5828 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6559622f3e402a8b215fb266d10472 (DEFAULT_TABLE table=test-table [id=0e47ad2eb4a144e291e471a986a5cb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:13.445225 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.392748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:36113 (local address 127.0.0.1:54750)
0807 06:49:13.417638 (+ 24890us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.417668 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.417936 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.421875 (+ 3939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.421879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.421902 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:13.422458 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.422472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.424573 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.424578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.427074 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.427087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.430053 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.430059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.430280 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.430356 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.430468 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.430638 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24681,"thread_start_us":26665,"threads_started":1}
I20260807 06:49:13.445808 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.393392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.27.54.66:36113)
0807 06:49:13.418573 (+ 25181us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.418584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.418621 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:49:13.418781 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.418792 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.419128 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.419393 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:13.422765 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.424372 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.428006 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.429512 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.429655 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.429875 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.431138 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.431315 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25009,"thread_start_us":24890,"threads_started":1}
I20260807 06:49:13.446269 5828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6559622f3e402a8b215fb266d10472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:13.449424 6062 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6559622f3e402a8b215fb266d10472 (DEFAULT_TABLE table=test-table [id=0e47ad2eb4a144e291e471a986a5cb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:13.450836 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6559622f3e402a8b215fb266d10472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:13.471448 6409 tablet_bootstrap.cc:492] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Bootstrap starting.
I20260807 06:49:13.477144 6409 tablet_bootstrap.cc:654] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:13.479177 6409 log.cc:824] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:13.484968 6410 tablet_bootstrap.cc:492] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Bootstrap starting.
I20260807 06:49:13.497087 6410 tablet_bootstrap.cc:654] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:13.499210 6410 log.cc:824] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:13.503983 6413 tablet_bootstrap.cc:492] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Bootstrap starting.
I20260807 06:49:13.506503 6409 tablet_bootstrap.cc:492] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: No bootstrap required, opened a new log
I20260807 06:49:13.506948 6409 ts_tablet_manager.cc:1397] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.014s
I20260807 06:49:13.508389 6410 tablet_bootstrap.cc:492] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: No bootstrap required, opened a new log
I20260807 06:49:13.509526 6410 ts_tablet_manager.cc:1397] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.007s
I20260807 06:49:13.510775 6413 tablet_bootstrap.cc:654] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:13.513273 6413 log.cc:824] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:13.519253 6409 raft_consensus.cc:348] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.519973 6409 raft_consensus.cc:374] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:13.520164 6409 raft_consensus.cc:729] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b11280d810438b9c5efb9800a5e40a, State: Initialized, Role: FOLLOWER
I20260807 06:49:13.521129 6409 consensus_queue.cc:260] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.521744 6410 raft_consensus.cc:348] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.522418 6410 raft_consensus.cc:374] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:13.522603 6410 raft_consensus.cc:729] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec855a623ec4065b270632de521ca74, State: Initialized, Role: FOLLOWER
I20260807 06:49:13.523672 6410 consensus_queue.cc:260] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.524083 6413 tablet_bootstrap.cc:492] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: No bootstrap required, opened a new log
I20260807 06:49:13.524487 6413 ts_tablet_manager.cc:1397] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260807 06:49:13.531915 6160 heartbeater.cc:503] Master 127.27.54.126:42485 was elected leader, sending a full tablet report...
I20260807 06:49:13.532279 6410 ts_tablet_manager.cc:1428] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20260807 06:49:13.543637 6361 heartbeater.cc:503] Master 127.27.54.126:42485 was elected leader, sending a full tablet report...
I20260807 06:49:13.544301 6409 ts_tablet_manager.cc:1428] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Time spent starting tablet: real 0.037s user 0.017s sys 0.012s
I20260807 06:49:13.543992 6413 raft_consensus.cc:348] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.544533 6413 raft_consensus.cc:374] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:13.544646 6413 raft_consensus.cc:729] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c49bf3489f4112ab6bde0f7612c3b9, State: Initialized, Role: FOLLOWER
I20260807 06:49:13.547076 6413 consensus_queue.cc:260] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.550031 6413 ts_tablet_manager.cc:1428] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Time spent starting tablet: real 0.025s user 0.014s sys 0.004s
I20260807 06:49:13.629914 6416 raft_consensus.cc:482] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:13.630265 6416 raft_consensus.cc:504] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.632247 6416 leader_election.cc:288] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931), 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
W20260807 06:49:13.637148 6165 tablet.cc:2369] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:49:13.643982 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.633175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:47761 (local address 127.27.54.65:35161)
0807 06:49:13.633447 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.633457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.635196 (+ 1739us) server_negotiation.cc:407] Connection header received
0807 06:49:13.635418 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.635422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.635485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.635587 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:13.636435 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.637668 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.641003 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.642773 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.643008 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.643120 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.643436 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.643624 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 06:49:13.644054 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.632710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:35161 (local address 127.27.54.66:47761)
0807 06:49:13.635058 (+ 2348us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.635073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.635263 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.635604 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.635614 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.635642 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:13.636199 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.636211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.637894 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.637898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.640759 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.640777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.643044 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.643051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.643230 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.643276 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.643400 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.643550 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2223,"mutex_wait_us":201,"thread_start_us":2058,"threads_started":1}
I20260807 06:49:13.645290 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.633140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:54283 (local address 127.27.54.67:44931)
0807 06:49:13.633406 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.633416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.634610 (+ 1194us) server_negotiation.cc:407] Connection header received
0807 06:49:13.634938 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.634947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.635047 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.635124 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:13.636368 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.637669 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.642808 (+ 5139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.644231 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.644406 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.644503 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.644900 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.645031 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:49:13.645453 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.632435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:44931 (local address 127.27.54.66:54283)
0807 06:49:13.634440 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:13.634467 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:13.634698 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:13.635427 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:13.635436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:13.635471 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:13.636166 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.636186 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.637890 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.637896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.640759 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:13.640777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.644584 (+ 3807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.644590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:13.644720 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.644744 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:49:13.644861 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 06:49:13.644994 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"mutex_wait_us":75,"thread_start_us":1539,"threads_started":1}
I20260807 06:49:13.645776 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6559622f3e402a8b215fb266d10472" candidate_uuid: "5ec855a623ec4065b270632de521ca74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" is_pre_election: true
I20260807 06:49:13.646569 5859 raft_consensus.cc:2454] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec855a623ec4065b270632de521ca74 in term 0.
I20260807 06:49:13.646611 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6559622f3e402a8b215fb266d10472" candidate_uuid: "5ec855a623ec4065b270632de521ca74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b11280d810438b9c5efb9800a5e40a" is_pre_election: true
I20260807 06:49:13.647301 6297 raft_consensus.cc:2454] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec855a623ec4065b270632de521ca74 in term 0.
I20260807 06:49:13.647485 5985 leader_election.cc:302] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c49bf3489f4112ab6bde0f7612c3b9, 5ec855a623ec4065b270632de521ca74; no voters:
I20260807 06:49:13.648066 6416 raft_consensus.cc:2781] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:13.648264 6416 raft_consensus.cc:482] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:13.648391 6416 raft_consensus.cc:3037] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:13.652932 6416 raft_consensus.cc:504] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.653951 6416 leader_election.cc:288] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [CANDIDATE]: Term 1 election: Requested vote from peers b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931), 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:13.654539 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6559622f3e402a8b215fb266d10472" candidate_uuid: "5ec855a623ec4065b270632de521ca74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c49bf3489f4112ab6bde0f7612c3b9"
I20260807 06:49:13.654579 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6559622f3e402a8b215fb266d10472" candidate_uuid: "5ec855a623ec4065b270632de521ca74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b11280d810438b9c5efb9800a5e40a"
I20260807 06:49:13.654826 5859 raft_consensus.cc:3037] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:13.654886 6297 raft_consensus.cc:3037] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:13.657447 5859 raft_consensus.cc:2454] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec855a623ec4065b270632de521ca74 in term 1.
I20260807 06:49:13.657997 5985 leader_election.cc:302] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c49bf3489f4112ab6bde0f7612c3b9, 5ec855a623ec4065b270632de521ca74; no voters:
I20260807 06:49:13.658414 6416 raft_consensus.cc:2781] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:13.659224 6416 raft_consensus.cc:686] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec855a623ec4065b270632de521ca74, State: Running, Role: LEADER
I20260807 06:49:13.659406 6297 raft_consensus.cc:2454] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec855a623ec4065b270632de521ca74 in term 1.
I20260807 06:49:13.660161 6416 consensus_queue.cc:237] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } }
I20260807 06:49:13.666325 4477 catalog_manager.cc:5495] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec855a623ec4065b270632de521ca74 (127.27.54.66). New cstate: current_term: 1 leader_uuid: "5ec855a623ec4065b270632de521ca74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } health_report { overall_health: UNKNOWN } } }
W20260807 06:49:13.696367 5931 tablet.cc:2369] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:49:13.741676 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.734257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.27.54.66:36113)
0807 06:49:13.734460 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.734473 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.734584 (+ 111us) server_negotiation.cc:407] Connection header received
0807 06:49:13.734880 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.734884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.734969 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.735116 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:13.735658 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.736461 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.739145 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.739635 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.739745 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.740476 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:13.741005 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:13.741120 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.741294 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.741370 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:49:13.752983 6297 raft_consensus.cc:1261] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Refusing update from remote peer 5ec855a623ec4065b270632de521ca74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:13.752986 5859 raft_consensus.cc:1261] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Refusing update from remote peer 5ec855a623ec4065b270632de521ca74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:13.754133 6416 consensus_queue.cc:1035] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:13.754491 6421 consensus_queue.cc:1035] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:49:13.770669 6366 tablet.cc:2369] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:49:13.775888 6427 mvcc.cc:205] Tried to move back new op lower bound from 7315805608959533056 to 7315805608596537344. Current Snapshot: MvccSnapshot[applied={T|T < 7315805608959533056}]
I20260807 06:49:13.778990 6431 mvcc.cc:205] Tried to move back new op lower bound from 7315805608959533056 to 7315805608596537344. Current Snapshot: MvccSnapshot[applied={T|T < 7315805608959533056}]
I20260807 06:49:13.783869 6430 mvcc.cc:205] Tried to move back new op lower bound from 7315805608959533056 to 7315805608596537344. Current Snapshot: MvccSnapshot[applied={T|T < 7315805608959533056}]
I20260807 06:49:13.827610 4477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:13.827934 4477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:13.835718 4477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8d3f22699b24f46b6bf3d716f39b696: Sending DeleteTablet for 3 replicas of tablet 1f6559622f3e402a8b215fb266d10472
I20260807 06:49:13.837563 6062 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6559622f3e402a8b215fb266d10472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:13 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:54750
I20260807 06:49:13.837741 5828 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6559622f3e402a8b215fb266d10472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:13 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:37612
I20260807 06:49:13.840232 6270 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6559622f3e402a8b215fb266d10472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:13 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47644
I20260807 06:49:13.848367 6444 tablet_replica.cc:330] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: stopping tablet replica
I20260807 06:49:13.848968 6444 raft_consensus.cc:2229] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:13.849563 6444 raft_consensus.cc:2258] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:13.850924 6442 tablet_replica.cc:330] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: stopping tablet replica
I20260807 06:49:13.852398 6442 raft_consensus.cc:2229] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:13.853745 6442 raft_consensus.cc:2258] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:13.855566 6441 tablet_replica.cc:330] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: stopping tablet replica
I20260807 06:49:13.856668 6441 raft_consensus.cc:2229] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:49:13.857223 6444 ts_tablet_manager.cc:1905] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:13.861146 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 4432
I20260807 06:49:13.862548 4522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:13.866983 6444 ts_tablet_manager.cc:1918] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:13.867324 6444 log.cc:1196] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/1f6559622f3e402a8b215fb266d10472
I20260807 06:49:13.868054 6444 ts_tablet_manager.cc:1939] T 1f6559622f3e402a8b215fb266d10472 P b5b11280d810438b9c5efb9800a5e40a: Deleting consensus metadata
I20260807 06:49:13.871498 6442 ts_tablet_manager.cc:1905] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:13.871894 6441 raft_consensus.cc:2258] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:13.878219 6441 ts_tablet_manager.cc:1905] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:13.880681 6442 ts_tablet_manager.cc:1918] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:13.881019 6442 log.cc:1196] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/1f6559622f3e402a8b215fb266d10472
I20260807 06:49:13.881850 6442 ts_tablet_manager.cc:1939] T 1f6559622f3e402a8b215fb266d10472 P 12c49bf3489f4112ab6bde0f7612c3b9: Deleting consensus metadata
I20260807 06:49:13.886561 6441 ts_tablet_manager.cc:1918] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:13.886885 6441 log.cc:1196] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/1f6559622f3e402a8b215fb266d10472
I20260807 06:49:13.887603 6441 ts_tablet_manager.cc:1939] T 1f6559622f3e402a8b215fb266d10472 P 5ec855a623ec4065b270632de521ca74: Deleting consensus metadata
I20260807 06:49:13.896799 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.841829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.27.54.125:44405)
0807 06:49:13.854905 (+ 13076us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.854916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.875223 (+ 20307us) server_negotiation.cc:407] Connection header received
0807 06:49:13.875325 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.875331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.875437 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.879312 (+ 3875us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.879564 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.880729 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.883209 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.884001 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.884153 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.888715 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.888723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.888731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.888791 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.893205 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.894217 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.894223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.894227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.894423 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.894929 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.894934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.894938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.895360 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.895630 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.896098 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.896374 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12844,"thread_start_us":12690,"threads_started":1}
I20260807 06:49:13.916931 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:13.860234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.27.54.124:38271)
0807 06:49:13.876452 (+ 16218us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:13.876459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:13.896933 (+ 20474us) server_negotiation.cc:407] Connection header received
0807 06:49:13.897225 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:13.897230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:13.897335 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:13.897419 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:13.899526 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.900739 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.902921 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:13.903689 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:13.903852 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:13.908046 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:13.908054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:13.908059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:13.908109 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:13.912581 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.913646 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.913653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.913657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.913829 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:13.914343 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:13.914349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:13.914353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:13.914802 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:13.915081 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:13.916243 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0807 06:49:13.916507 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16052,"thread_start_us":18902,"threads_started":1}
I20260807 06:49:14.030436 4453 catalog_manager.cc:4841] TS 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113): tablet 1f6559622f3e402a8b215fb266d10472 (table test-table [id=0e47ad2eb4a144e291e471a986a5cb3b]) successfully deleted
I20260807 06:49:14.031761 4452 catalog_manager.cc:4841] TS b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931): tablet 1f6559622f3e402a8b215fb266d10472 (table test-table [id=0e47ad2eb4a144e291e471a986a5cb3b]) successfully deleted
W20260807 06:49:14.032014 4452 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.54.67:44931, user_credentials={real_user=slave}} blocked reactor thread for 161677us
I20260807 06:49:14.032678 4451 catalog_manager.cc:4841] TS 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161): tablet 1f6559622f3e402a8b215fb266d10472 (table test-table [id=0e47ad2eb4a144e291e471a986a5cb3b]) successfully deleted
I20260807 06:49:14.033129 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 4432
I20260807 06:49:14.092475 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:14.069995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.27.54.125:44405)
0807 06:49:14.075017 (+ 5022us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:14.075024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:14.075049 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:49:14.075840 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:14.075846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:14.075930 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:14.076038 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:14.078844 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:14.079908 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:14.081756 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:14.082213 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:14.082369 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:14.085472 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:14.085504 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:14.085508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:14.085548 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:14.089212 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:14.089919 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:14.089924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:14.089928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:14.090077 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:14.090554 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:14.090559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:14.090562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:14.091042 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:14.091336 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:14.091853 (+ 517us) server_negotiation.cc:299] Negotiation successful
0807 06:49:14.092115 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4779}
I20260807 06:49:14.093415 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:14.070033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.27.54.124:38271)
0807 06:49:14.070298 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:14.070308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:14.075236 (+ 4928us) server_negotiation.cc:407] Connection header received
0807 06:49:14.075557 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:14.075565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:14.075655 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:14.075740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:14.078905 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:14.079624 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:14.081673 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:14.082312 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:14.082443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:14.086204 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:14.086214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:14.086222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:14.086271 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:14.090236 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:14.091027 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:14.091033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:14.091037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:14.091222 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:14.091733 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:14.091738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:14.091741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:14.092106 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:14.092355 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:14.092953 (+ 598us) server_negotiation.cc:299] Negotiation successful
0807 06:49:14.093174 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:49:14.797909 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:14.796936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:37169)
0807 06:49:14.797505 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:14.797674 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":295,"threads_started":1}
W20260807 06:49:14.798987 5927 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:42485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:14.810877 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:14.810020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:43727)
0807 06:49:14.810435 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:14.810652 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
W20260807 06:49:14.811331 6361 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:42485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:14.814427 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:14.813580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:36639)
0807 06:49:14.814056 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:14.814215 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
W20260807 06:49:14.814850 6160 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:42485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:15.338264 6473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8d3f22699b24f46b6bf3d716f39b696)
I20260807 06:49:15.338161 6470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8d3f22699b24f46b6bf3d716f39b696)
I20260807 06:49:15.338765 6473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:15.338845 6470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:15.341364 6470 leader_election.cc:288] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125:44405)
I20260807 06:49:15.350335 6473 leader_election.cc:288] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124:38271)
I20260807 06:49:15.362036 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.342928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42754)
0807 06:49:15.356043 (+ 13115us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.356360 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12884,"thread_start_us":16321,"threads_started":1}
W20260807 06:49:15.366158 5645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:15.367384 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.350376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42770)
0807 06:49:15.362398 (+ 12022us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.366488 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11466,"thread_start_us":11282,"threads_started":1}
W20260807 06:49:15.368018 5359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:15.381143 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.380574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42784)
0807 06:49:15.380781 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.380910 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 06:49:15.382040 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.380574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42788)
0807 06:49:15.381513 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.381783 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":772}
W20260807 06:49:15.383029 5645 leader_election.cc:334] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
W20260807 06:49:15.383324 5359 leader_election.cc:334] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:15.389298 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.361628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:38271 (local address 127.0.0.1:60630)
0807 06:49:15.367227 (+ 5599us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.367252 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:15.367430 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:15.373165 (+ 5735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:15.373174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:15.373209 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:15.376321 (+ 3112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:15.376349 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.380850 (+ 4501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.380857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:15.383633 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:15.383652 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.386631 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.386687 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:15.387727 (+ 1040us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:15.388026 (+ 299us) client_negotiation.cc:769] Sending connection context
0807 06:49:15.388543 (+ 517us) client_negotiation.cc:240] Negotiation successful
0807 06:49:15.388824 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5448,"mutex_wait_us":22,"thread_start_us":5143,"threads_started":1}
I20260807 06:49:15.389393 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.360448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.27.54.124:38271)
0807 06:49:15.363947 (+ 3499us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:15.363989 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:15.367357 (+ 3368us) server_negotiation.cc:407] Connection header received
0807 06:49:15.371732 (+ 4375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:15.371793 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:15.372505 (+ 712us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:15.372726 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:15.377412 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.380717 (+ 3305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.383877 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.385888 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.386258 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:15.386596 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:15.388687 (+ 2091us) server_negotiation.cc:299] Negotiation successful
0807 06:49:15.388876 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":2945,"spinlock_wait_cycles":9600,"thread_start_us":799,"threads_started":1}
I20260807 06:49:15.391752 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.343793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:44405 (local address 127.0.0.1:43686)
0807 06:49:15.363875 (+ 20082us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.363988 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:15.367886 (+ 3898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:15.371232 (+ 3346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:15.371280 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:15.371573 (+ 293us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:15.373174 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:15.373204 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.380887 (+ 7683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.380894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:15.384236 (+ 3342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:15.384252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.389297 (+ 5045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.389303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:15.389530 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:15.389572 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 06:49:15.389768 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 06:49:15.390883 (+ 1115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19511,"mutex_wait_us":317,"spinlock_wait_cycles":41728,"thread_start_us":24410,"threads_started":1}
I20260807 06:49:15.392037 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02ef9d212a134bf88539c91ff2ea6089" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" is_pre_election: true
I20260807 06:49:15.393612 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.343875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.27.54.125:44405)
0807 06:49:15.357724 (+ 13849us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:15.357796 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:15.364710 (+ 6914us) server_negotiation.cc:407] Connection header received
0807 06:49:15.369007 (+ 4297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:15.369041 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:15.369715 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:15.370406 (+ 691us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:15.374267 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.380676 (+ 6409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.384438 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:15.389053 (+ 4615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:15.389212 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:15.389402 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:15.390981 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0807 06:49:15.391496 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1419,"server-negotiator.queue_time_us":12450,"thread_start_us":13635,"threads_started":1}
I20260807 06:49:15.394162 5678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ef9d212a134bf88539c91ff2ea6089 in term 1.
I20260807 06:49:15.394790 5399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02ef9d212a134bf88539c91ff2ea6089" is_pre_election: true
I20260807 06:49:15.394889 5362 leader_election.cc:302] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ef9d212a134bf88539c91ff2ea6089, 7e85fbd644cc40f1a0d7cfd0147970f7; no voters: d8d3f22699b24f46b6bf3d716f39b696
I20260807 06:49:15.395098 5399 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e85fbd644cc40f1a0d7cfd0147970f7 in term 1.
I20260807 06:49:15.395349 6473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 06:49:15.395498 6473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8d3f22699b24f46b6bf3d716f39b696)
I20260807 06:49:15.395704 6473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:49:15.396073 5647 leader_election.cc:302] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ef9d212a134bf88539c91ff2ea6089, 7e85fbd644cc40f1a0d7cfd0147970f7; no voters: d8d3f22699b24f46b6bf3d716f39b696
I20260807 06:49:15.396512 6470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 06:49:15.396833 6470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8d3f22699b24f46b6bf3d716f39b696)
I20260807 06:49:15.397115 6470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:49:15.402909 6473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:15.405541 6473 leader_election.cc:288] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 2 election: Requested vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124:38271)
I20260807 06:49:15.406577 6470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:15.407533 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.405788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42798)
0807 06:49:15.406619 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.407129 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550}
I20260807 06:49:15.407864 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02ef9d212a134bf88539c91ff2ea6089" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
I20260807 06:49:15.408344 5678 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 02ef9d212a134bf88539c91ff2ea6089 in current term 2: Already voted for candidate 7e85fbd644cc40f1a0d7cfd0147970f7 in this term.
I20260807 06:49:15.410547 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.408533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42810)
0807 06:49:15.409202 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.409792 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 06:49:15.409996 5399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02ef9d212a134bf88539c91ff2ea6089"
I20260807 06:49:15.411424 5399 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e85fbd644cc40f1a0d7cfd0147970f7 in current term 2: Already voted for candidate 02ef9d212a134bf88539c91ff2ea6089 in this term.
I20260807 06:49:15.411420 6470 leader_election.cc:288] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 election: Requested vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 02ef9d212a134bf88539c91ff2ea6089 (127.27.54.125:44405)
I20260807 06:49:15.411795 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.409570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42820)
0807 06:49:15.410111 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.410535 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260807 06:49:15.412971 5359 leader_election.cc:334] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:15.413241 5359 leader_election.cc:302] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02ef9d212a134bf88539c91ff2ea6089; no voters: 7e85fbd644cc40f1a0d7cfd0147970f7, d8d3f22699b24f46b6bf3d716f39b696
I20260807 06:49:15.413511 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.413143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:42832)
0807 06:49:15.413335 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.413403 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 06:49:15.413638 6473 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 06:49:15.413686 5645 leader_election.cc:334] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:15.413832 5645 leader_election.cc:302] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e85fbd644cc40f1a0d7cfd0147970f7; no voters: 02ef9d212a134bf88539c91ff2ea6089, d8d3f22699b24f46b6bf3d716f39b696
I20260807 06:49:15.414377 6470 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 06:49:15.800999 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.799916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:51461)
0807 06:49:15.800535 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.800709 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":326,"threads_started":1}
I20260807 06:49:15.812837 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.812059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:47517)
0807 06:49:15.812553 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.812653 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":275,"threads_started":1}
I20260807 06:49:15.816569 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:15.815597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:35451)
0807 06:49:15.816282 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:15.816384 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":416,"threads_started":1}
I20260807 06:49:16.802909 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:16.801881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:39137)
0807 06:49:16.802457 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:16.802573 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":288,"threads_started":1}
I20260807 06:49:16.814543 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:16.813764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:53029)
0807 06:49:16.814222 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:16.814345 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260807 06:49:16.818197 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:16.817491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:55195)
0807 06:49:16.817943 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:16.818029 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":253,"threads_started":1}
I20260807 06:49:17.078491 6496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:17.078862 6496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:17.079746 6496 leader_election.cc:288] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124:38271)
I20260807 06:49:17.080417 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02ef9d212a134bf88539c91ff2ea6089" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" is_pre_election: true
I20260807 06:49:17.080720 5678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ef9d212a134bf88539c91ff2ea6089 in term 2.
I20260807 06:49:17.080914 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.079947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43080)
0807 06:49:17.080527 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.080679 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":316,"threads_started":1}
I20260807 06:49:17.081220 5362 leader_election.cc:302] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ef9d212a134bf88539c91ff2ea6089, 7e85fbd644cc40f1a0d7cfd0147970f7; no voters:
I20260807 06:49:17.081542 6496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 06:49:17.081702 6496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:17.081823 6496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:49:17.082556 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.082143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43096)
0807 06:49:17.082352 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.082424 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 06:49:17.082799 5359 leader_election.cc:334] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:17.084568 6496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:17.085163 6496 leader_election.cc:288] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 3 election: Requested vote from peers d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485), 7e85fbd644cc40f1a0d7cfd0147970f7 (127.27.54.124:38271)
I20260807 06:49:17.085820 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.085356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43098)
0807 06:49:17.085538 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.085635 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 06:49:17.085717 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02ef9d212a134bf88539c91ff2ea6089" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7"
I20260807 06:49:17.085973 5678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:49:17.086972 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.086523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43104)
0807 06:49:17.086715 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.086839 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 06:49:17.087162 5359 leader_election.cc:334] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
I20260807 06:49:17.088712 5678 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ef9d212a134bf88539c91ff2ea6089 in term 3.
I20260807 06:49:17.089279 5362 leader_election.cc:302] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ef9d212a134bf88539c91ff2ea6089, 7e85fbd644cc40f1a0d7cfd0147970f7; no voters: d8d3f22699b24f46b6bf3d716f39b696
I20260807 06:49:17.089632 6496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 06:49:17.090090 6496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [term 3 LEADER]: Becoming Leader. State: Replica: 02ef9d212a134bf88539c91ff2ea6089, State: Running, Role: LEADER
I20260807 06:49:17.090728 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } }
I20260807 06:49:17.093672 6500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02ef9d212a134bf88539c91ff2ea6089. Latest consensus state: current_term: 3 leader_uuid: "02ef9d212a134bf88539c91ff2ea6089" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:17.094107 6500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:17.095028 6502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:17.101553 6502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:17.102553 6502 catalog_manager.cc:1259] Loaded cluster ID: b4ece8fc898c4f84b0e66249daa1d237
I20260807 06:49:17.102635 6502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:17.104007 6502 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:17.104967 6502 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Loaded TSK: 0
I20260807 06:49:17.106316 6502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:49:17.189222 5384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:49:17.197918 5384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:17.203892 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.203303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43134)
0807 06:49:17.203540 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.203660 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260807 06:49:17.204092 5678 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [term 3 FOLLOWER]: Refusing update from remote peer 02ef9d212a134bf88539c91ff2ea6089: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260807 06:49:17.204993 6500 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 06:49:17.205170 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.204747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43138)
0807 06:49:17.204979 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.205052 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:49:17.205415 5359 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 -> Peer d8d3f22699b24f46b6bf3d716f39b696 (127.27.54.126:42485): Couldn't send request to peer d8d3f22699b24f46b6bf3d716f39b696. Status: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 06:49:17.208240 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02ef9d212a134bf88539c91ff2ea6089. Latest consensus state: current_term: 3 leader_uuid: "02ef9d212a134bf88539c91ff2ea6089" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:17.208563 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:17.210318 6500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "02ef9d212a134bf88539c91ff2ea6089" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:17.210629 6500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:17.211287 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "02ef9d212a134bf88539c91ff2ea6089" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8d3f22699b24f46b6bf3d716f39b696" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42485 } } peers { permanent_uuid: "02ef9d212a134bf88539c91ff2ea6089" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 44405 } attrs { promote: false } } peers { permanent_uuid: "7e85fbd644cc40f1a0d7cfd0147970f7" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 38271 } attrs { promote: false } } }
I20260807 06:49:17.211596 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e85fbd644cc40f1a0d7cfd0147970f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:49:17.234462 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.221717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:36113 (local address 127.0.0.1:54762)
0807 06:49:17.221934 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.221974 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:17.222177 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:17.223086 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:17.223091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:17.223124 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:17.223514 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.223533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.225750 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.225756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.228688 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.228715 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.231560 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.231566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.231832 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.231892 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 06:49:17.232049 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 06:49:17.232283 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 06:49:17.236289 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.221891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.27.54.66:36113)
0807 06:49:17.222620 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.222628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.222689 (+ 61us) server_negotiation.cc:407] Connection header received
0807 06:49:17.222798 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.222804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.222899 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.223698 (+ 799us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:17.223982 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.225547 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.230398 (+ 4851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.231358 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.231524 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.231665 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.235464 (+ 3799us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.235716 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":409,"threads_started":1}
I20260807 06:49:17.237386 6062 tablet_service.cc:1467] Processing CreateTablet for tablet 91d2e8eceebf42c5a8938878295ada1e (DEFAULT_TABLE table=test-table [id=d9786399b3b047f49e36a550b6461dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:17.238129 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d2e8eceebf42c5a8938878295ada1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:17.244694 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.223035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:35161 (local address 127.0.0.1:37622)
0807 06:49:17.235040 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.235066 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:17.235220 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:17.238004 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:17.238011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:17.238075 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:17.238331 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.238358 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.240264 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.240270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.242067 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.242083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.243822 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.243828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.243987 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.244022 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 06:49:17.244130 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 06:49:17.244281 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11842,"mutex_wait_us":42,"spinlock_wait_cycles":13056,"thread_start_us":15486,"threads_started":1}
I20260807 06:49:17.244702 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.223475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.27.54.65:35161)
0807 06:49:17.237578 (+ 14103us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.237585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.237608 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:49:17.237705 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.237710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.237789 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.238183 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:17.238973 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.240106 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.242274 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.243614 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.243782 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.243909 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.244184 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.244339 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13966,"thread_start_us":13840,"threads_started":1}
I20260807 06:49:17.245745 5828 tablet_service.cc:1467] Processing CreateTablet for tablet 91d2e8eceebf42c5a8938878295ada1e (DEFAULT_TABLE table=test-table [id=d9786399b3b047f49e36a550b6461dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:17.245885 6518 tablet_bootstrap.cc:492] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Bootstrap starting.
I20260807 06:49:17.246258 5828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d2e8eceebf42c5a8938878295ada1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:17.246618 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.223633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:44931 (local address 127.0.0.1:47656)
0807 06:49:17.236119 (+ 12486us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.236141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:17.236284 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:17.237777 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:17.237786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:17.237824 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:17.238748 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.238761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.240941 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.240946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.242866 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.242878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.245899 (+ 3021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.245909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.246097 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.246129 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:49:17.246217 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 06:49:17.246357 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12148,"thread_start_us":12272,"threads_started":1}
I20260807 06:49:17.247318 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.223915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.27.54.67:44931)
0807 06:49:17.237330 (+ 13415us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.237337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.237379 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:49:17.237466 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.237475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.237584 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.239431 (+ 1847us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:17.239658 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.240763 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.243034 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.245480 (+ 2446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.245620 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.245738 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.246810 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.246984 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13234,"thread_start_us":13108,"threads_started":1}
I20260807 06:49:17.248090 6270 tablet_service.cc:1467] Processing CreateTablet for tablet 91d2e8eceebf42c5a8938878295ada1e (DEFAULT_TABLE table=test-table [id=d9786399b3b047f49e36a550b6461dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:17.248165 6518 tablet_bootstrap.cc:654] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:17.248528 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d2e8eceebf42c5a8938878295ada1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:17.254365 6522 tablet_bootstrap.cc:492] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Bootstrap starting.
I20260807 06:49:17.255129 6524 tablet_bootstrap.cc:492] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Bootstrap starting.
I20260807 06:49:17.256170 6518 tablet_bootstrap.cc:492] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: No bootstrap required, opened a new log
I20260807 06:49:17.256364 6518 ts_tablet_manager.cc:1397] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260807 06:49:17.256412 6522 tablet_bootstrap.cc:654] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:17.257048 6524 tablet_bootstrap.cc:654] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:17.257236 6518 raft_consensus.cc:348] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.257560 6518 raft_consensus.cc:374] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:17.257707 6518 raft_consensus.cc:729] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec855a623ec4065b270632de521ca74, State: Initialized, Role: FOLLOWER
I20260807 06:49:17.257998 6518 consensus_queue.cc:260] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.259560 6522 tablet_bootstrap.cc:492] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: No bootstrap required, opened a new log
I20260807 06:49:17.259716 6522 ts_tablet_manager.cc:1397] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:49:17.260586 6524 tablet_bootstrap.cc:492] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: No bootstrap required, opened a new log
I20260807 06:49:17.260735 6524 ts_tablet_manager.cc:1397] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260807 06:49:17.260727 6518 ts_tablet_manager.cc:1428] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260807 06:49:17.261036 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.260332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:46825)
0807 06:49:17.260747 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.260866 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260807 06:49:17.260938 6522 raft_consensus.cc:348] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.261196 6522 raft_consensus.cc:374] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:17.261296 6522 raft_consensus.cc:729] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c49bf3489f4112ab6bde0f7612c3b9, State: Initialized, Role: FOLLOWER
I20260807 06:49:17.261620 6522 consensus_queue.cc:260] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.261649 6524 raft_consensus.cc:348] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.261891 6524 raft_consensus.cc:374] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:17.261961 6524 raft_consensus.cc:729] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b11280d810438b9c5efb9800a5e40a, State: Initialized, Role: FOLLOWER
I20260807 06:49:17.262267 6524 consensus_queue.cc:260] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.266912 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.266141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:34115)
0807 06:49:17.266454 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.266579 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260807 06:49:17.270025 6163 heartbeater.cc:503] Master 127.27.54.125:44405 was elected leader, sending a full tablet report...
I20260807 06:49:17.273661 6522 ts_tablet_manager.cc:1428] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260807 06:49:17.273908 6524 ts_tablet_manager.cc:1428] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260807 06:49:17.274988 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.274394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:34229)
0807 06:49:17.274710 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.274810 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260807 06:49:17.276153 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.275632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:49283)
0807 06:49:17.275872 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.275956 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260807 06:49:17.276750 6364 heartbeater.cc:503] Master 127.27.54.125:44405 was elected leader, sending a full tablet report...
I20260807 06:49:17.277302 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.276731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:44769)
0807 06:49:17.276986 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.277080 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260807 06:49:17.281006 5930 heartbeater.cc:503] Master 127.27.54.125:44405 was elected leader, sending a full tablet report...
I20260807 06:49:17.281248 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.280473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:49977)
0807 06:49:17.280902 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.280999 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314}
I20260807 06:49:17.469834 6531 raft_consensus.cc:482] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:17.470091 6531 raft_consensus.cc:504] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.471992 6531 leader_election.cc:288] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113), 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:17.479429 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.472378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:59459 (local address 127.27.54.66:36113)
0807 06:49:17.472579 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.472586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.472712 (+ 126us) server_negotiation.cc:407] Connection header received
0807 06:49:17.472990 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.472999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.473097 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.473220 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:17.473837 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.474923 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.477150 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.478128 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.478267 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.478332 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.478952 (+ 620us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.479144 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:49:17.479439 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.472230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:36113 (local address 127.27.54.67:59459)
0807 06:49:17.472559 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.472587 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:17.472775 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:17.473252 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:17.473256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:17.473282 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:17.473619 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.473633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.475076 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.475083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.476965 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.476979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.478361 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.478368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.478760 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.478791 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 06:49:17.478884 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 06:49:17.479004 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":94,"spinlock_wait_cycles":21248}
I20260807 06:49:17.479621 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.472627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:51767 (local address 127.27.54.65:35161)
0807 06:49:17.472848 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.472856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.473395 (+ 539us) server_negotiation.cc:407] Connection header received
0807 06:49:17.473619 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.473625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.473722 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.473826 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:17.474436 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.475403 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.477381 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.478344 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.478542 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.478631 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.479173 (+ 542us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.479325 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 06:49:17.480036 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.472289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:35161 (local address 127.27.54.67:51767)
0807 06:49:17.473264 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.473285 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:17.473450 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:17.473869 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:17.473874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:17.473905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:17.474258 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.474272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.475565 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.475569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.477179 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:17.477198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.478561 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.478566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:17.478951 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.478990 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 06:49:17.479096 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 06:49:17.479264 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":129,"spinlock_wait_cycles":1280,"thread_start_us":565,"threads_started":1}
I20260807 06:49:17.480669 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d2e8eceebf42c5a8938878295ada1e" candidate_uuid: "b5b11280d810438b9c5efb9800a5e40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec855a623ec4065b270632de521ca74" is_pre_election: true
I20260807 06:49:17.480778 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d2e8eceebf42c5a8938878295ada1e" candidate_uuid: "b5b11280d810438b9c5efb9800a5e40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" is_pre_election: true
I20260807 06:49:17.481060 5859 raft_consensus.cc:2454] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b11280d810438b9c5efb9800a5e40a in term 0.
I20260807 06:49:17.481127 6091 raft_consensus.cc:2454] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b11280d810438b9c5efb9800a5e40a in term 0.
I20260807 06:49:17.481753 6206 leader_election.cc:302] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c49bf3489f4112ab6bde0f7612c3b9, b5b11280d810438b9c5efb9800a5e40a; no voters:
I20260807 06:49:17.482195 6531 raft_consensus.cc:2781] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:17.482350 6531 raft_consensus.cc:482] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:17.482467 6531 raft_consensus.cc:3037] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:17.484952 6531 raft_consensus.cc:504] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.485654 6531 leader_election.cc:288] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [CANDIDATE]: Term 1 election: Requested vote from peers 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113), 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161)
I20260807 06:49:17.486109 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d2e8eceebf42c5a8938878295ada1e" candidate_uuid: "b5b11280d810438b9c5efb9800a5e40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec855a623ec4065b270632de521ca74"
I20260807 06:49:17.486153 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d2e8eceebf42c5a8938878295ada1e" candidate_uuid: "b5b11280d810438b9c5efb9800a5e40a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c49bf3489f4112ab6bde0f7612c3b9"
I20260807 06:49:17.486373 5859 raft_consensus.cc:3037] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:17.486366 6091 raft_consensus.cc:3037] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:17.489877 5859 raft_consensus.cc:2454] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b11280d810438b9c5efb9800a5e40a in term 1.
I20260807 06:49:17.489877 6091 raft_consensus.cc:2454] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b11280d810438b9c5efb9800a5e40a in term 1.
I20260807 06:49:17.490432 6208 leader_election.cc:302] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12c49bf3489f4112ab6bde0f7612c3b9, 5ec855a623ec4065b270632de521ca74, b5b11280d810438b9c5efb9800a5e40a; no voters:
I20260807 06:49:17.490863 6531 raft_consensus.cc:2781] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:17.491343 6531 raft_consensus.cc:686] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 1 LEADER]: Becoming Leader. State: Replica: b5b11280d810438b9c5efb9800a5e40a, State: Running, Role: LEADER
I20260807 06:49:17.491886 6531 consensus_queue.cc:237] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } }
I20260807 06:49:17.492524 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.492075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:45113)
0807 06:49:17.492263 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.492380 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 06:49:17.495680 5383 catalog_manager.cc:5495] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b11280d810438b9c5efb9800a5e40a (127.27.54.67). New cstate: current_term: 1 leader_uuid: "b5b11280d810438b9c5efb9800a5e40a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b11280d810438b9c5efb9800a5e40a" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44931 } health_report { overall_health: HEALTHY } } }
I20260807 06:49:17.539924 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.534024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.27.54.67:44931)
0807 06:49:17.534255 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.534262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.534282 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:49:17.534499 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.534503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.534601 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.534792 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:17.535123 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.535939 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.537336 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.537794 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.537938 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.538408 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:17.539163 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:17.539280 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.539540 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.539676 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:49:17.544639 5859 raft_consensus.cc:1261] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Refusing update from remote peer b5b11280d810438b9c5efb9800a5e40a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:17.545526 6091 raft_consensus.cc:1261] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Refusing update from remote peer b5b11280d810438b9c5efb9800a5e40a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:17.546386 6531 consensus_queue.cc:1035] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c49bf3489f4112ab6bde0f7612c3b9" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:17.547067 6531 consensus_queue.cc:1035] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec855a623ec4065b270632de521ca74" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:17.549165 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.548646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:56539)
0807 06:49:17.548861 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.548972 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 06:49:17.550470 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.549917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:42817)
0807 06:49:17.550181 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.550286 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260807 06:49:17.556646 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.556004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.65:46891)
0807 06:49:17.556329 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.556431 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260807 06:49:17.557601 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.556977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.66:49097)
0807 06:49:17.557190 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.557414 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260807 06:49:17.564853 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.564277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:36089)
0807 06:49:17.564524 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.564648 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260807 06:49:17.566144 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.565563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.27.54.67:46069)
0807 06:49:17.565860 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.565975 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260807 06:49:17.573935 5384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:17.574240 5384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:17.580483 5384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02ef9d212a134bf88539c91ff2ea6089: Sending DeleteTablet for 3 replicas of tablet 91d2e8eceebf42c5a8938878295ada1e
I20260807 06:49:17.581677 5828 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d2e8eceebf42c5a8938878295ada1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:17 UTC) from {username='oryx', principal='oryx/127.27.54.125@KRBTEST.COM'} at 127.0.0.1:37622
I20260807 06:49:17.581682 6062 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d2e8eceebf42c5a8938878295ada1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:17 UTC) from {username='oryx', principal='oryx/127.27.54.125@KRBTEST.COM'} at 127.0.0.1:54762
I20260807 06:49:17.582302 6270 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d2e8eceebf42c5a8938878295ada1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:17 UTC) from {username='oryx', principal='oryx/127.27.54.125@KRBTEST.COM'} at 127.0.0.1:47656
I20260807 06:49:17.588932 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 5631
I20260807 06:49:17.593879 6553 tablet_replica.cc:330] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: stopping tablet replica
I20260807 06:49:17.594686 6549 tablet_replica.cc:330] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: stopping tablet replica
I20260807 06:49:17.594610 6553 raft_consensus.cc:2229] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:49:17.594949 6549 raft_consensus.cc:2229] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:17.595306 6549 raft_consensus.cc:2258] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:17.595671 6553 raft_consensus.cc:2258] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:17.597122 6549 ts_tablet_manager.cc:1905] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:17.605217 6549 ts_tablet_manager.cc:1918] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:17.605404 6549 log.cc:1196] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/91d2e8eceebf42c5a8938878295ada1e
I20260807 06:49:17.605868 6551 tablet_replica.cc:330] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: stopping tablet replica
I20260807 06:49:17.606002 6549 ts_tablet_manager.cc:1939] T 91d2e8eceebf42c5a8938878295ada1e P 5ec855a623ec4065b270632de521ca74: Deleting consensus metadata
I20260807 06:49:17.606252 6551 raft_consensus.cc:2229] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:17.606936 6551 raft_consensus.cc:2258] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:17.607460 6553 ts_tablet_manager.cc:1905] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:17.608870 6551 ts_tablet_manager.cc:1905] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:17.615527 6553 ts_tablet_manager.cc:1918] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:17.615710 6553 log.cc:1196] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/91d2e8eceebf42c5a8938878295ada1e
I20260807 06:49:17.616245 6553 ts_tablet_manager.cc:1939] T 91d2e8eceebf42c5a8938878295ada1e P b5b11280d810438b9c5efb9800a5e40a: Deleting consensus metadata
I20260807 06:49:17.617733 5361 catalog_manager.cc:4841] TS b5b11280d810438b9c5efb9800a5e40a (127.27.54.67:44931): tablet 91d2e8eceebf42c5a8938878295ada1e (table test-table [id=d9786399b3b047f49e36a550b6461dba]) successfully deleted
I20260807 06:49:17.616080 6551 ts_tablet_manager.cc:1918] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:17.619059 6551 log.cc:1196] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/91d2e8eceebf42c5a8938878295ada1e
I20260807 06:49:17.619578 6551 ts_tablet_manager.cc:1939] T 91d2e8eceebf42c5a8938878295ada1e P 12c49bf3489f4112ab6bde0f7612c3b9: Deleting consensus metadata
I20260807 06:49:17.620072 5362 catalog_manager.cc:4841] TS 5ec855a623ec4065b270632de521ca74 (127.27.54.66:36113): tablet 91d2e8eceebf42c5a8938878295ada1e (table test-table [id=d9786399b3b047f49e36a550b6461dba]) successfully deleted
I20260807 06:49:17.620872 5359 catalog_manager.cc:4841] TS 12c49bf3489f4112ab6bde0f7612c3b9 (127.27.54.65:35161): tablet 91d2e8eceebf42c5a8938878295ada1e (table test-table [id=d9786399b3b047f49e36a550b6461dba]) successfully deleted
I20260807 06:49:17.633849 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.590664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.27.54.65:35161)
0807 06:49:17.590918 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.590928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.605576 (+ 14648us) server_negotiation.cc:407] Connection header received
0807 06:49:17.605863 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.605869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.605977 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.606090 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:17.610741 (+ 4651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.611970 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.617966 (+ 5996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.618633 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.618759 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.625763 (+ 7004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:17.625800 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:17.625829 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:17.625881 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:17.630111 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:17.631174 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:17.631185 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:17.631188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:17.631386 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:17.631895 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:17.631900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:17.631904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:17.632427 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:17.632798 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.633174 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.633476 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 06:49:17.634619 5917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:17.734055 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 5631
I20260807 06:49:17.763252 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 5944
I20260807 06:49:17.767457 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.762446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43144)
0807 06:49:17.767099 (+ 4653us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.767234 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4521,"thread_start_us":4411,"threads_started":1}
I20260807 06:49:17.772290 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.771831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:42485 (local address 127.0.0.1:43146)
0807 06:49:17.772014 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:17.772119 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:42485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 06:49:17.781502 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.764476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.27.54.66:36113)
0807 06:49:17.764662 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.764668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.764701 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:49:17.764779 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.764786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.764877 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.765010 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:17.767352 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.768647 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.771262 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.771862 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.771970 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.775161 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:17.775177 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:17.775193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:17.775230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:17.778347 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:17.779276 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:17.779293 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:17.779296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:17.779444 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:17.779866 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:17.779873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:17.779881 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:17.780269 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:17.780609 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.780916 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.781154 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:49:17.782181 6144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:17.907970 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 5944
I20260807 06:49:17.953173 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 6181
I20260807 06:49:17.971663 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:17.954080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.27.54.67:44931)
0807 06:49:17.954264 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:17.954269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:17.954310 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:49:17.954532 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:17.954536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:17.954646 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:17.954803 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:17.956966 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.957925 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.959869 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:17.960330 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:17.960446 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:17.964311 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:17.964331 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:17.964355 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:17.964403 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:17.967965 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:17.968890 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:17.968898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:17.968900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:17.969068 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:17.969595 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:17.969609 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:17.969611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:17.970083 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:17.970361 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:17.970831 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 06:49:17.971032 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:49:17.972483 6353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:18.118122 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 6181
I20260807 06:49:18.147965 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 5177
I20260807 06:49:18.149175 5425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:18.275432 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 5177
I20260807 06:49:18.310117 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 5330
I20260807 06:49:18.311401 5709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:18.432087 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 5330
2026-08-07T06:49:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12664 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:49:18 dist-test-slave-3kk6 krb5kdc[6570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:49:18 dist-test-slave-3kk6 krb5kdc[6570](info): set up 2 sockets
Aug 07 06:49:18 dist-test-slave-3kk6 krb5kdc[6570](info): commencing operation
krb5kdc: starting...
W20260807 06:49:19.798063 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:49:19 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab.
Entry for principal oryx/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab.
I20260807 06:49:20.118531 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41245
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab
--principal=oryx/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:20.560992 6588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:20.561389 6588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:20.561575 6588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:20.569031 6588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:20.569227 6588 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 06:49:20.569312 6588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:20.569414 6588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:20.569514 6588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:20.569617 6588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:20.580000 6588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:41245
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.126.keytab
--principal=oryx/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:20.581538 6588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:20.583020 6588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:20.596659 6594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:20.596735 6595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:20.596729 6598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:20.597129 6588 server_base.cc:1034] running on GCE node
I20260807 06:49:20.598417 6588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:20.600010 6588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:20.601284 6588 hybrid_clock.cc:648] HybridClock initialized: now 1786085360601213 us; error 40 us; skew 500 ppm
I20260807 06:49:20.601662 6588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:20 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085360, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:20.605193 6588 init.cc:377] Logged in from keytab as oryx/127.27.54.126@KRBTEST.COM (short username oryx)
I20260807 06:49:20.607329 6588 webserver.cc:460] Webserver started at http://127.27.54.126:40763/ using document root <none> and password file <none>
I20260807 06:49:20.608151 6588 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:20.608302 6588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:20.608697 6588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:20.612102 6588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "d4575be679374e2b90b977553dc6722c"
format_stamp: "Formatted at 2026-08-07 06:49:20 on dist-test-slave-3kk6"
server_key: "1317ac27bf75a08ccc5101a38945d1db"
server_key_iv: "dca74df8f28ff51c30811af873348989"
server_key_version: "encryptionkey@0"
I20260807 06:49:20.612998 6588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "d4575be679374e2b90b977553dc6722c"
format_stamp: "Formatted at 2026-08-07 06:49:20 on dist-test-slave-3kk6"
server_key: "1317ac27bf75a08ccc5101a38945d1db"
server_key_iv: "dca74df8f28ff51c30811af873348989"
server_key_version: "encryptionkey@0"
I20260807 06:49:20.618939 6588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 06:49:20.622809 6610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:20.624215 6588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:20.624449 6588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "d4575be679374e2b90b977553dc6722c"
format_stamp: "Formatted at 2026-08-07 06:49:20 on dist-test-slave-3kk6"
server_key: "1317ac27bf75a08ccc5101a38945d1db"
server_key_iv: "dca74df8f28ff51c30811af873348989"
server_key_version: "encryptionkey@0"
I20260807 06:49:20.624692 6588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:20.641110 6588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:20.646157 6588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:20.646494 6588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:20.670920 6588 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:41245
I20260807 06:49:20.670943 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:41245 every 8 connection(s)
I20260807 06:49:20.672466 6588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:20.676513 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:20.682088 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 6588
I20260807 06:49:20.682374 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:20.683265 27865 external_mini_cluster.cc:1428] Setting key 393d860d955f8aa6e67b2b89a36ffbf1
I20260807 06:49:20.689136 6700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Bootstrap starting.
I20260807 06:49:20.692480 6700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:20.693624 6700 log.cc:824] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:20.696813 6700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: No bootstrap required, opened a new log
Aug 07 06:49:20 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:20.707490 6700 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4575be679374e2b90b977553dc6722c" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41245 } }
I20260807 06:49:20.707986 6700 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:20.708101 6700 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4575be679374e2b90b977553dc6722c, State: Initialized, Role: FOLLOWER
I20260807 06:49:20.708781 6700 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4575be679374e2b90b977553dc6722c" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41245 } }
I20260807 06:49:20.709102 6700 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:20.709267 6700 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:20.709312 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:20.685814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.27.54.126:41245)
0807 06:49:20.686818 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:20.686831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:20.686878 (+ 47us) server_negotiation.cc:407] Connection header received
0807 06:49:20.688804 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:20.688829 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:20.689299 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:20.689729 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:20.691419 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:20.692827 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:20.694300 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:20.695187 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:20.695400 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:20.699926 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:20.699939 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:20.699964 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:20.700011 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:20.704330 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:20.705131 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:20.705141 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:20.705150 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:20.705400 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:20.705855 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:20.705860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:20.705863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:20.706641 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:20.706968 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:20.707415 (+ 447us) server_negotiation.cc:299] Negotiation successful
0807 06:49:20.707877 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":391,"threads_started":1}
I20260807 06:49:20.709493 6700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:20.712061 6700 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4575be679374e2b90b977553dc6722c" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41245 } }
I20260807 06:49:20.712592 6700 leader_election.cc:302] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4575be679374e2b90b977553dc6722c; no voters:
I20260807 06:49:20.713176 6700 leader_election.cc:288] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:20.713332 6708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:20.714277 6708 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [term 1 LEADER]: Becoming Leader. State: Replica: d4575be679374e2b90b977553dc6722c, State: Running, Role: LEADER
I20260807 06:49:20.714979 6708 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4575be679374e2b90b977553dc6722c" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41245 } }
I20260807 06:49:20.716356 6700 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:20.717962 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4575be679374e2b90b977553dc6722c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4575be679374e2b90b977553dc6722c" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41245 } } }
I20260807 06:49:20.717979 6709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4575be679374e2b90b977553dc6722c. Latest consensus state: current_term: 1 leader_uuid: "d4575be679374e2b90b977553dc6722c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4575be679374e2b90b977553dc6722c" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41245 } } }
I20260807 06:49:20.718528 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:20.718528 6709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:20.722754 6721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:20.727928 6721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:20.737427 6721 catalog_manager.cc:1347] Generated new cluster ID: e06813e57deb4523abef2a5e3b73b72b
I20260807 06:49:20.737573 6721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:20.764751 6721 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:20.766827 6721 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:20.780197 6721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Generated new TSK 0
I20260807 06:49:20.781219 6721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab.
Entry for principal oryx/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab.
I20260807 06:49:20.850688 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41245
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:21.210538 6745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:21.211007 6745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:21.211195 6745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:21.219280 6745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:21.219439 6745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:21.219600 6745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:49:21.219753 6745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:21.230983 6745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:41245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:21.232424 6745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:21.233902 6745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:21.248026 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:21.248003 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:21.247946 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:21.249125 6745 server_base.cc:1034] running on GCE node
I20260807 06:49:21.249828 6745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:21.251078 6745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:21.252354 6745 hybrid_clock.cc:648] HybridClock initialized: now 1786085361252281 us; error 51 us; skew 500 ppm
I20260807 06:49:21.252697 6745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:21 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085361, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:21.256256 6745 init.cc:377] Logged in from keytab as oryx/127.27.54.65@KRBTEST.COM (short username oryx)
I20260807 06:49:21.257781 6745 webserver.cc:460] Webserver started at http://127.27.54.65:33439/ using document root <none> and password file <none>
I20260807 06:49:21.258548 6745 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:21.258752 6745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:21.259137 6745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:21.262279 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "d0fb4917e8d64999bcc270395ba1f9b2"
format_stamp: "Formatted at 2026-08-07 06:49:21 on dist-test-slave-3kk6"
server_key: "b601deaa847e6b57b2a4e2c86c21e835"
server_key_iv: "671e600e4de01e87f8968dc8068d66df"
server_key_version: "encryptionkey@0"
I20260807 06:49:21.263188 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "d0fb4917e8d64999bcc270395ba1f9b2"
format_stamp: "Formatted at 2026-08-07 06:49:21 on dist-test-slave-3kk6"
server_key: "b601deaa847e6b57b2a4e2c86c21e835"
server_key_iv: "671e600e4de01e87f8968dc8068d66df"
server_key_version: "encryptionkey@0"
I20260807 06:49:21.268725 6745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:49:21.272596 6764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:21.274083 6745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:49:21.274303 6745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "d0fb4917e8d64999bcc270395ba1f9b2"
format_stamp: "Formatted at 2026-08-07 06:49:21 on dist-test-slave-3kk6"
server_key: "b601deaa847e6b57b2a4e2c86c21e835"
server_key_iv: "671e600e4de01e87f8968dc8068d66df"
server_key_version: "encryptionkey@0"
I20260807 06:49:21.274565 6745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:21.299697 6745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:21.304729 6745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:21.305084 6745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:21.308549 6745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:21.308676 6745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:21.308845 6745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:21.308938 6745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:49:21 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085361, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:21.338276 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.309117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:44989 (local address 127.27.54.126:41245)
0807 06:49:21.310148 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:21.310154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:21.311113 (+ 959us) server_negotiation.cc:407] Connection header received
0807 06:49:21.313620 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:21.313627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:21.313729 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:21.313877 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:21.316778 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.318144 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.321504 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.322369 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.322558 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.328524 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:21.328534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:21.328541 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:21.328596 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:21.333243 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.334197 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.334207 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.334213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.334416 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.335273 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.335280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.335287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.335765 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:21.336003 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:21.337315 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0807 06:49:21.337687 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":787,"threads_started":1}
I20260807 06:49:21.339195 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.309915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.27.54.65:44989)
0807 06:49:21.310911 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:21.310968 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:21.313274 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:21.314200 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:21.314218 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:21.315373 (+ 1155us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:21.316455 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.316481 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.318396 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.318409 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.321199 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.321253 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.322766 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.322782 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.323258 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.324298 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:21.324338 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:21.328198 (+ 3860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:21.333438 (+ 5240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.333450 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.333476 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.334021 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.334603 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.334617 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.334620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.335048 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.335974 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:21.335987 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:21.336740 (+ 753us) client_negotiation.cc:769] Sending connection context
0807 06:49:21.337151 (+ 411us) client_negotiation.cc:240] Negotiation successful
0807 06:49:21.337716 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":411,"threads_started":1}
I20260807 06:49:21.366315 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.346195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:58619 (local address 127.27.54.126:41245)
0807 06:49:21.346476 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:21.346485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:21.347959 (+ 1474us) server_negotiation.cc:407] Connection header received
0807 06:49:21.348134 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:21.348144 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:21.348285 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:21.348445 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:21.349521 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.351039 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.353784 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.354808 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.354992 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.357737 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:21.357748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:21.357756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:21.357810 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:21.362237 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.363578 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.363586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.363591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.363812 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.364244 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.364250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.364252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.364715 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:21.364915 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:21.365520 (+ 605us) server_negotiation.cc:299] Negotiation successful
0807 06:49:21.365857 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 06:49:21.366400 6745 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:44713
I20260807 06:49:21.366436 6944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:44713 every 8 connection(s)
I20260807 06:49:21.366549 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.346101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.27.54.65:58619)
0807 06:49:21.347788 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:21.347814 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:21.347981 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:21.348470 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:21.348473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:21.349008 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:21.349333 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.349343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.351279 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.351291 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.353527 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.353552 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.355110 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.355123 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.355353 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.356365 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:21.356398 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:21.357455 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:21.362456 (+ 5001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.362467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.362473 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.363399 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.363970 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.363974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.363976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.364116 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.364925 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:21.364931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:21.365174 (+ 243us) client_negotiation.cc:769] Sending connection context
0807 06:49:21.365381 (+ 207us) client_negotiation.cc:240] Negotiation successful
0807 06:49:21.365890 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1529,"thread_start_us":1369,"threads_started":1}
I20260807 06:49:21.368072 6745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:21.370887 6945 heartbeater.cc:347] Connected to a master server at 127.27.54.126:41245
I20260807 06:49:21.371153 6945 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:21.371857 6945 heartbeater.cc:511] Master 127.27.54.126:41245 requested a full tablet report, sending...
I20260807 06:49:21.373716 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 6745
I20260807 06:49:21.373983 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:21.374080 6632 ts_manager.cc:194] Registered new tserver with Master: d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713)
I20260807 06:49:21.374809 27865 external_mini_cluster.cc:1428] Setting key 9c2bf480ae54417d988ec8e2460bc21f
I20260807 06:49:21.375908 6632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:44989
WARNING: no policy specified for oryx/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab.
Entry for principal oryx/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab.
I20260807 06:49:21.443001 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41245
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:21.808524 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:21.808925 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:21.809118 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:21.817298 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:21.817483 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:21.817636 6953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:49:21.817786 6953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:21.829284 6953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:41245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:21.830927 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:21.832451 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:21.847821 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:21.847836 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:21.847904 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:21.848910 6953 server_base.cc:1034] running on GCE node
I20260807 06:49:21.849560 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:21.850585 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:21.851891 6953 hybrid_clock.cc:648] HybridClock initialized: now 1786085361851795 us; error 62 us; skew 500 ppm
I20260807 06:49:21.852247 6953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:21 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085361, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:21.856207 6953 init.cc:377] Logged in from keytab as oryx/127.27.54.66@KRBTEST.COM (short username oryx)
I20260807 06:49:21.857998 6953 webserver.cc:460] Webserver started at http://127.27.54.66:41111/ using document root <none> and password file <none>
I20260807 06:49:21.858879 6953 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:21.859030 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:21.859405 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:21.862823 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "8dc7b396e5814956a0b6034c6b45e158"
format_stamp: "Formatted at 2026-08-07 06:49:21 on dist-test-slave-3kk6"
server_key: "86cfac4b0fcf78dc825f8d5625831293"
server_key_iv: "aad0e7675aa30912786729acf91e3cc6"
server_key_version: "encryptionkey@0"
I20260807 06:49:21.863775 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "8dc7b396e5814956a0b6034c6b45e158"
format_stamp: "Formatted at 2026-08-07 06:49:21 on dist-test-slave-3kk6"
server_key: "86cfac4b0fcf78dc825f8d5625831293"
server_key_iv: "aad0e7675aa30912786729acf91e3cc6"
server_key_version: "encryptionkey@0"
I20260807 06:49:21.869904 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 06:49:21.874094 6971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:21.875674 6953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:21.875888 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "8dc7b396e5814956a0b6034c6b45e158"
format_stamp: "Formatted at 2026-08-07 06:49:21 on dist-test-slave-3kk6"
server_key: "86cfac4b0fcf78dc825f8d5625831293"
server_key_iv: "aad0e7675aa30912786729acf91e3cc6"
server_key_version: "encryptionkey@0"
I20260807 06:49:21.876164 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:21.891466 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:21.896315 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:21.896672 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:21.899969 6953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:21.900098 6953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:21.900280 6953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:21.900410 6953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 07 06:49:21 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085361, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:21.927867 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.900552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:36761 (local address 127.27.54.126:41245)
0807 06:49:21.901514 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:21.901519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:21.902316 (+ 797us) server_negotiation.cc:407] Connection header received
0807 06:49:21.904551 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:21.904557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:21.904683 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:21.904799 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:21.907390 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.908837 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.911913 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.912523 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.912703 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.919212 (+ 6509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:21.919222 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:21.919228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:21.919288 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:21.922884 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.924275 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.924284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.924288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.924484 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.925250 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.925255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.925259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.925683 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:21.925881 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:21.927205 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0807 06:49:21.927510 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":716,"threads_started":1}
I20260807 06:49:21.930204 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.901242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.27.54.66:36761)
0807 06:49:21.902122 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:21.902168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:21.904226 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:21.905198 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:21.905207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:21.906019 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:21.907001 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.907029 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.909081 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.909088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.911689 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.911710 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.912925 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.912937 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.913346 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.914347 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:21.914378 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:21.918768 (+ 4390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:21.923087 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.923098 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.923121 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.923999 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.924643 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.924652 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.924656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.925095 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.925962 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:21.925968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:21.926580 (+ 612us) client_negotiation.cc:769] Sending connection context
0807 06:49:21.928245 (+ 1665us) client_negotiation.cc:240] Negotiation successful
0807 06:49:21.928871 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":383,"threads_started":1}
I20260807 06:49:21.954901 6953 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:33063
I20260807 06:49:21.954945 7150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:33063 every 8 connection(s)
I20260807 06:49:21.956617 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:21.959299 7151 heartbeater.cc:347] Connected to a master server at 127.27.54.126:41245
I20260807 06:49:21.959641 7151 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:21.960465 7151 heartbeater.cc:511] Master 127.27.54.126:41245 requested a full tablet report, sending...
I20260807 06:49:21.962334 6631 ts_manager.cc:194] Registered new tserver with Master: 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063)
I20260807 06:49:21.963884 6631 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:36761
I20260807 06:49:21.966300 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 6953
I20260807 06:49:21.966529 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:21.967140 27865 external_mini_cluster.cc:1428] Setting key ace5866125e552f6a875a77c0fa938b9
I20260807 06:49:21.968168 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.951019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:49517 (local address 127.27.54.126:41245)
0807 06:49:21.951256 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:21.951263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:21.952251 (+ 988us) server_negotiation.cc:407] Connection header received
0807 06:49:21.952540 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:21.952546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:21.952640 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:21.952767 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:21.954473 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.955682 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.957303 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.958057 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.958214 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.960867 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:21.960873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:21.960877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:21.960916 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:21.964752 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.965540 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.965546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.965550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.965740 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:21.966265 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:21.966271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:21.966274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:21.966805 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:21.966977 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:21.967524 (+ 547us) server_negotiation.cc:299] Negotiation successful
0807 06:49:21.967779 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 06:49:21.968166 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:21.951163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.27.54.66:49517)
0807 06:49:21.952054 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:21.952087 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:21.952328 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:21.952819 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:21.952822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:21.953552 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:21.954239 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.954262 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.955878 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.955883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.957054 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:21.957069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:21.958353 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:21.958360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:21.958576 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:21.959467 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:21.959500 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:21.960526 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:21.964908 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.964914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.964919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.965390 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.965910 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:21.965917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:21.965924 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:21.966118 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:21.966973 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:21.966979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:21.967196 (+ 217us) client_negotiation.cc:769] Sending connection context
0807 06:49:21.967394 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 06:49:21.967749 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":492,"threads_started":1}
WARNING: no policy specified for oryx/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab.
Entry for principal oryx/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab.
I20260807 06:49:22.036569 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41245
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:22.379788 6945 heartbeater.cc:503] Master 127.27.54.126:41245 was elected leader, sending a full tablet report...
W20260807 06:49:22.406929 7162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:22.407240 7162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:22.407473 7162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:22.415160 7162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:22.415313 7162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:22.415514 7162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 06:49:22.415654 7162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:22.426779 7162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.54.126:41245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:22.428234 7162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:22.429733 7162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:22.444402 7169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:22.444475 7171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:22.444437 7168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:22.445268 7162 server_base.cc:1034] running on GCE node
I20260807 06:49:22.445986 7162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:22.447072 7162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:22.448318 7162 hybrid_clock.cc:648] HybridClock initialized: now 1786085362448253 us; error 38 us; skew 500 ppm
I20260807 06:49:22.448654 7162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:22 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085362, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:22.458186 7162 init.cc:377] Logged in from keytab as oryx/127.27.54.67@KRBTEST.COM (short username oryx)
I20260807 06:49:22.460086 7162 webserver.cc:460] Webserver started at http://127.27.54.67:45459/ using document root <none> and password file <none>
I20260807 06:49:22.460907 7162 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:22.461020 7162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:22.461418 7162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:22.464854 7162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "bfaefa9bf4394f328ea027a2c8787975"
format_stamp: "Formatted at 2026-08-07 06:49:22 on dist-test-slave-3kk6"
server_key: "2343a0a61ad5b89af34b8fc98a417095"
server_key_iv: "c94dbfff1a616a799bba2a10c936adaa"
server_key_version: "encryptionkey@0"
I20260807 06:49:22.465792 7162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "bfaefa9bf4394f328ea027a2c8787975"
format_stamp: "Formatted at 2026-08-07 06:49:22 on dist-test-slave-3kk6"
server_key: "2343a0a61ad5b89af34b8fc98a417095"
server_key_iv: "c94dbfff1a616a799bba2a10c936adaa"
server_key_version: "encryptionkey@0"
I20260807 06:49:22.471673 7162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:49:22.475510 7182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:22.476977 7162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:22.477164 7162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "bfaefa9bf4394f328ea027a2c8787975"
format_stamp: "Formatted at 2026-08-07 06:49:22 on dist-test-slave-3kk6"
server_key: "2343a0a61ad5b89af34b8fc98a417095"
server_key_iv: "c94dbfff1a616a799bba2a10c936adaa"
server_key_version: "encryptionkey@0"
I20260807 06:49:22.477420 7162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:22.506526 7162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:22.511819 7162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:22.512198 7162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:22.515513 7162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:22.515641 7162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:22.515802 7162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:22.515892 7162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:49:22 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085362, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:22.546967 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:22.516247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:53157 (local address 127.27.54.126:41245)
0807 06:49:22.517012 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:22.517018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:22.518378 (+ 1360us) server_negotiation.cc:407] Connection header received
0807 06:49:22.521299 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:22.521310 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:22.521424 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:22.521533 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:22.525113 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.526380 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.529422 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.530146 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.530345 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:22.536910 (+ 6565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:22.536921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:22.536928 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:22.536986 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:22.540992 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:22.541943 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:22.541950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:22.541956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:22.542150 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:22.542736 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:22.542742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:22.542745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:22.543102 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:22.543290 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:22.546009 (+ 2719us) server_negotiation.cc:299] Negotiation successful
0807 06:49:22.546332 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":500,"threads_started":1}
I20260807 06:49:22.547281 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:22.516922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.27.54.67:53157)
0807 06:49:22.518007 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:22.518090 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:22.520735 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:22.522344 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:22.522353 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:22.523573 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:22.524826 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:22.524866 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.526600 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.526605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:22.529207 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:22.529227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.530555 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.530571 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:22.531044 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:22.532127 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:22.532172 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:22.536500 (+ 4328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:22.541206 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:22.541216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:22.541237 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:22.541765 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:22.542272 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:22.542278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:22.542282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:22.542551 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:22.543298 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:22.543309 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:22.544025 (+ 716us) client_negotiation.cc:769] Sending connection context
0807 06:49:22.544481 (+ 456us) client_negotiation.cc:240] Negotiation successful
0807 06:49:22.545334 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":400,"threads_started":1}
I20260807 06:49:22.580384 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:22.559093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.27.54.67:48387)
0807 06:49:22.563335 (+ 4242us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:22.563373 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:22.563653 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:22.564117 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:22.564122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:22.564907 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:22.565461 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:22.565479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.566692 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.566700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:22.568391 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:22.568411 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.571048 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.571059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:22.571235 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:22.572290 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:22.572321 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:22.573547 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:22.577401 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:22.577408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:22.577412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:22.577881 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:22.578327 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:22.578332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:22.578336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:22.578482 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:22.579208 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:22.579215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:22.579418 (+ 203us) client_negotiation.cc:769] Sending connection context
0807 06:49:22.579581 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 06:49:22.579957 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3983,"thread_start_us":3776,"threads_started":1}
I20260807 06:49:22.580907 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:22.558714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:48387 (local address 127.27.54.126:41245)
0807 06:49:22.558913 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:22.558918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:22.563597 (+ 4679us) server_negotiation.cc:407] Connection header received
0807 06:49:22.563820 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:22.563824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:22.563909 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:22.564070 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:22.565677 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.566450 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.568601 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.569287 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.569443 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:22.573880 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:22.573886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:22.573889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:22.573927 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:22.577216 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:22.578054 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:22.578059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:22.578061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:22.578199 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:22.578615 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:22.578625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:22.578627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:22.579089 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:22.579261 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:22.579862 (+ 601us) server_negotiation.cc:299] Negotiation successful
0807 06:49:22.580256 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:49:22.584197 7162 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:33149
I20260807 06:49:22.584308 7379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:33149 every 8 connection(s)
I20260807 06:49:22.585983 7162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:22.588303 7381 heartbeater.cc:347] Connected to a master server at 127.27.54.126:41245
I20260807 06:49:22.588570 7381 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:22.589306 7381 heartbeater.cc:511] Master 127.27.54.126:41245 requested a full tablet report, sending...
I20260807 06:49:22.590525 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 7162
I20260807 06:49:22.590775 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:49:22.591037 6632 ts_manager.cc:194] Registered new tserver with Master: bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:22.591315 27865 external_mini_cluster.cc:1428] Setting key 09698a8c30ff92b0d961a5e3a06b5abf
I20260807 06:49:22.592245 6632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:53157
I20260807 06:49:22.595000 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 06:49:22 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 06:49:22 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:22 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:22.967844 7151 heartbeater.cc:503] Master 127.27.54.126:41245 was elected leader, sending a full tablet report...
I20260807 06:49:23.596200 7381 heartbeater.cc:503] Master 127.27.54.126:41245 was elected leader, sending a full tablet report...
W20260807 06:49:23.782617 7393 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.122s user 0.000s sys 0.023s
W20260807 06:49:23.784137 7199 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:22.653921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.27.54.126:41245)
0807 06:49:22.654186 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:22.654194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:22.654479 (+ 285us) server_negotiation.cc:407] Connection header received
0807 06:49:22.654777 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:22.654782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:22.654854 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:22.654972 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:22.656540 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.657305 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.658717 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:22.659167 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:22.659336 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.783885 (+1124549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53226: BlockingRecv error: recv got EOF from 127.0.0.1:53226 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 07 06:49:23 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:23.803421 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.788396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.27.54.126:41245)
0807 06:49:23.788599 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:23.788607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:23.789001 (+ 394us) server_negotiation.cc:407] Connection header received
0807 06:49:23.789239 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:23.789244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:23.789323 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:23.789440 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:23.790820 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.791532 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.792877 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.793301 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.793447 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.797756 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:23.797761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:23.797765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:23.797803 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:23.800579 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:23.801304 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:23.801312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:23.801321 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:23.801457 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:23.801861 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:23.801871 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:23.801873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:23.802157 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:23.802442 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:23.802858 (+ 416us) server_negotiation.cc:299] Negotiation successful
0807 06:49:23.803113 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:49:23.807894 6632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:49:23.815423 6632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:23.858104 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.829353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:33149 (local address 127.0.0.1:47672)
0807 06:49:23.847308 (+ 17955us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:23.847339 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:23.847533 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:23.849810 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:23.849822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:23.849886 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:23.850893 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:23.850912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.852591 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.852597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:23.855377 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:23.855397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.857001 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.857008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:23.857201 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.857269 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 06:49:23.857416 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 06:49:23.857647 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17760,"mutex_wait_us":5,"thread_start_us":20873,"threads_started":1}
I20260807 06:49:23.858846 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.829058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44713 (local address 127.0.0.1:51630)
0807 06:49:23.843044 (+ 13986us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:23.843119 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:23.843412 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:23.850983 (+ 7571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:23.850990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:23.851024 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:23.851500 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:23.851513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.853437 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.853445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:23.855516 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:23.855538 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.857580 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.857587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:23.857768 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.857807 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 06:49:23.857925 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:49:23.858072 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12576,"thread_start_us":12925,"threads_started":1}
I20260807 06:49:23.859449 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.829389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.27.54.65:44713)
0807 06:49:23.850144 (+ 20755us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:23.850154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:23.850253 (+ 99us) server_negotiation.cc:407] Connection header received
0807 06:49:23.850368 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:23.850376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:23.850620 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:23.851726 (+ 1106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:23.851962 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.853295 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.855716 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.857323 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.858250 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.858621 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:23.858831 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 06:49:23.859016 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20581,"thread_start_us":20421,"threads_started":1}
I20260807 06:49:23.859648 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.830000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.27.54.67:33149)
0807 06:49:23.848106 (+ 18106us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:23.848120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:23.848163 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:49:23.848302 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:23.848315 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:23.848703 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:23.849608 (+ 905us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:23.851129 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.852432 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.855578 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.856717 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.856932 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.857184 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:23.857482 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 06:49:23.857665 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17929,"thread_start_us":17766,"threads_started":1}
I20260807 06:49:23.860713 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.830865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.27.54.66:33063)
0807 06:49:23.849581 (+ 18716us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:23.849593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:23.851107 (+ 1514us) server_negotiation.cc:407] Connection header received
0807 06:49:23.851475 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:23.851484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:23.851740 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:23.851911 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:23.852874 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.854393 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.857107 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.858930 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.859056 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.859616 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:23.860224 (+ 608us) server_negotiation.cc:299] Negotiation successful
0807 06:49:23.860397 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18510,"thread_start_us":18241,"threads_started":1}
I20260807 06:49:23.863669 6840 tablet_service.cc:1467] Processing CreateTablet for tablet 44f7c030991a4f9e97473ea34d9287bf (DEFAULT_TABLE table=test-table [id=05372b68a02a4de58c88a1af68b5290b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:23.863682 7271 tablet_service.cc:1467] Processing CreateTablet for tablet 44f7c030991a4f9e97473ea34d9287bf (DEFAULT_TABLE table=test-table [id=05372b68a02a4de58c88a1af68b5290b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:23.864488 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:23.830020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:33063 (local address 127.0.0.1:36886)
0807 06:49:23.850974 (+ 20954us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:23.850995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:23.851219 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:23.852198 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:23.852203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:23.852231 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:23.852654 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:23.852671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.854578 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.854583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:23.856877 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:23.856893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:23.859836 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:23.859845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:23.860014 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:23.860043 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:49:23.860140 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 06:49:23.860282 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20793,"thread_start_us":16907,"threads_started":1}
I20260807 06:49:23.865304 6840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f7c030991a4f9e97473ea34d9287bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:23.865319 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f7c030991a4f9e97473ea34d9287bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:23.867802 7043 tablet_service.cc:1467] Processing CreateTablet for tablet 44f7c030991a4f9e97473ea34d9287bf (DEFAULT_TABLE table=test-table [id=05372b68a02a4de58c88a1af68b5290b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:23.869028 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f7c030991a4f9e97473ea34d9287bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:23.882630 7421 tablet_bootstrap.cc:492] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Bootstrap starting.
I20260807 06:49:23.883291 7420 tablet_bootstrap.cc:492] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Bootstrap starting.
I20260807 06:49:23.884195 7422 tablet_bootstrap.cc:492] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Bootstrap starting.
I20260807 06:49:23.888087 7422 tablet_bootstrap.cc:654] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:23.888394 7421 tablet_bootstrap.cc:654] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:23.889183 7420 tablet_bootstrap.cc:654] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:23.889467 7422 log.cc:824] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:23.889756 7421 log.cc:824] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:23.890523 7420 log.cc:824] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:23.893415 7422 tablet_bootstrap.cc:492] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: No bootstrap required, opened a new log
I20260807 06:49:23.893860 7422 ts_tablet_manager.cc:1397] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260807 06:49:23.893906 7420 tablet_bootstrap.cc:492] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: No bootstrap required, opened a new log
I20260807 06:49:23.894187 7420 ts_tablet_manager.cc:1397] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260807 06:49:23.894356 7421 tablet_bootstrap.cc:492] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: No bootstrap required, opened a new log
I20260807 06:49:23.894627 7421 ts_tablet_manager.cc:1397] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260807 06:49:23.901850 7420 raft_consensus.cc:348] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:23.902472 7420 raft_consensus.cc:374] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:23.902618 7420 raft_consensus.cc:729] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Initialized, Role: FOLLOWER
I20260807 06:49:23.903398 7420 consensus_queue.cc:260] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:23.905987 7422 raft_consensus.cc:348] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:23.905987 7421 raft_consensus.cc:348] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:23.906503 7421 raft_consensus.cc:374] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:23.906580 7422 raft_consensus.cc:374] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:23.906695 7421 raft_consensus.cc:729] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Initialized, Role: FOLLOWER
I20260807 06:49:23.906790 7422 raft_consensus.cc:729] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc7b396e5814956a0b6034c6b45e158, State: Initialized, Role: FOLLOWER
I20260807 06:49:23.907687 7420 ts_tablet_manager.cc:1428] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260807 06:49:23.907594 7421 consensus_queue.cc:260] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:23.907594 7422 consensus_queue.cc:260] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:23.909888 7421 ts_tablet_manager.cc:1428] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260807 06:49:23.912046 7422 ts_tablet_manager.cc:1428] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
W20260807 06:49:23.960566 7152 tablet.cc:2369] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:49:24.089351 7382 tablet.cc:2369] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:49:24.122182 6946 tablet.cc:2369] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:49:24.150367 7429 raft_consensus.cc:482] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:24.150866 7429 raft_consensus.cc:504] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:24.152765 7429 leader_election.cc:288] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713), 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063)
I20260807 06:49:24.160177 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.153759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:51165 (local address 127.27.54.65:44713)
0807 06:49:24.153984 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.153993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.154388 (+ 395us) server_negotiation.cc:407] Connection header received
0807 06:49:24.154532 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.154538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.154635 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.154778 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:24.155486 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.156207 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.158276 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.159120 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.159336 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.159428 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.159724 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.159888 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:49:24.160198 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.153393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:58297 (local address 127.27.54.66:33063)
0807 06:49:24.153713 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.153720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.154051 (+ 331us) server_negotiation.cc:407] Connection header received
0807 06:49:24.154348 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.154357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.154464 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.154568 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:24.155338 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.156177 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.158273 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.159167 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.159369 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.159457 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.159760 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.159911 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260807 06:49:24.160380 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.153221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44713 (local address 127.27.54.67:51165)
0807 06:49:24.154268 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.154283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.154402 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.154816 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.154820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.154853 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:24.155300 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.155313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.156376 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.156381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.158092 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.158108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.159369 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.159376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.159548 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.159574 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.159692 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.159887 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":52,"thread_start_us":737,"threads_started":1}
I20260807 06:49:24.160454 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.153143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:33063 (local address 127.27.54.67:58297)
0807 06:49:24.153929 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.153952 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.154154 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.154586 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.154589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.154608 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:24.155130 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.155148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.156338 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.156346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.158092 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.158108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.159435 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.159441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.159585 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.159608 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.159719 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.159887 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":82,"thread_start_us":396,"threads_started":1}
I20260807 06:49:24.161717 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7c030991a4f9e97473ea34d9287bf" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" is_pre_election: true
I20260807 06:49:24.162273 6870 raft_consensus.cc:2454] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 0.
I20260807 06:49:24.162386 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7c030991a4f9e97473ea34d9287bf" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158" is_pre_election: true
I20260807 06:49:24.163030 7077 raft_consensus.cc:2454] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 0.
I20260807 06:49:24.163043 7186 leader_election.cc:302] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:24.163615 7429 raft_consensus.cc:2781] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:24.163770 7429 raft_consensus.cc:482] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:24.163877 7429 raft_consensus.cc:3037] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.166252 7429 raft_consensus.cc:504] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:24.167006 7429 leader_election.cc:288] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 election: Requested vote from peers d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713), 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063)
I20260807 06:49:24.167618 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7c030991a4f9e97473ea34d9287bf" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158"
I20260807 06:49:24.167618 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7c030991a4f9e97473ea34d9287bf" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb4917e8d64999bcc270395ba1f9b2"
I20260807 06:49:24.167876 6870 raft_consensus.cc:3037] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.167872 7077 raft_consensus.cc:3037] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.171355 6870 raft_consensus.cc:2454] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 1.
I20260807 06:49:24.171365 7077 raft_consensus.cc:2454] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 1.
I20260807 06:49:24.171952 7183 leader_election.cc:302] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc7b396e5814956a0b6034c6b45e158, bfaefa9bf4394f328ea027a2c8787975; no voters:
I20260807 06:49:24.172356 7429 raft_consensus.cc:2781] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:24.172863 7429 raft_consensus.cc:686] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 1 LEADER]: Becoming Leader. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Running, Role: LEADER
I20260807 06:49:24.173483 7429 consensus_queue.cc:237] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } }
I20260807 06:49:24.178534 6628 catalog_manager.cc:5495] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67). New cstate: current_term: 1 leader_uuid: "bfaefa9bf4394f328ea027a2c8787975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260807 06:49:24.235868 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.230323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.27.54.67:33149)
0807 06:49:24.230540 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.230547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.230676 (+ 129us) server_negotiation.cc:407] Connection header received
0807 06:49:24.230968 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.230972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.231065 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.231227 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:24.231597 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.232326 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.233698 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.234117 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.234255 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.234729 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:24.235267 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:24.235452 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.235507 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.235594 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:49:24.244665 7077 raft_consensus.cc:1261] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Refusing update from remote peer bfaefa9bf4394f328ea027a2c8787975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:24.244665 6870 raft_consensus.cc:1261] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Refusing update from remote peer bfaefa9bf4394f328ea027a2c8787975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:24.245792 7432 consensus_queue.cc:1035] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:24.246155 7429 consensus_queue.cc:1035] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:24.261672 7442 mvcc.cc:205] Tried to move back new op lower bound from 7315805651934494720 to 7315805651658829824. Current Snapshot: MvccSnapshot[applied={T|T < 7315805651934494720 or (T in {7315805651934494720})}]
I20260807 06:49:24.273512 6631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:24.273800 6631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:24.277427 6631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Sending DeleteTablet for 3 replicas of tablet 44f7c030991a4f9e97473ea34d9287bf
I20260807 06:49:24.278728 6840 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f7c030991a4f9e97473ea34d9287bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:24 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:51630
I20260807 06:49:24.278759 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f7c030991a4f9e97473ea34d9287bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:24 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:36886
I20260807 06:49:24.279567 7271 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f7c030991a4f9e97473ea34d9287bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:24 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47672
I20260807 06:49:24.280282 6631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:49:24.280990 6631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:24.287187 7455 tablet_replica.cc:330] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: stopping tablet replica
I20260807 06:49:24.287729 7455 raft_consensus.cc:2229] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:24.289578 6837 tablet_service.cc:1467] Processing CreateTablet for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca (DEFAULT_TABLE table=test-table [id=37702ed62d474be4a10514326fa7378f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:24.289666 7270 tablet_service.cc:1467] Processing CreateTablet for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca (DEFAULT_TABLE table=test-table [id=37702ed62d474be4a10514326fa7378f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:24.290320 7270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.290386 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.295420 7043 tablet_service.cc:1467] Processing CreateTablet for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca (DEFAULT_TABLE table=test-table [id=37702ed62d474be4a10514326fa7378f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:24.296288 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.299818 7421 tablet_bootstrap.cc:492] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Bootstrap starting.
I20260807 06:49:24.301712 7421 tablet_bootstrap.cc:654] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.309331 7455 raft_consensus.cc:2258] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:24.310748 7420 tablet_bootstrap.cc:492] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Bootstrap starting.
I20260807 06:49:24.312999 7420 tablet_bootstrap.cc:654] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.313812 7458 tablet_replica.cc:330] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: stopping tablet replica
I20260807 06:49:24.314304 7422 tablet_bootstrap.cc:492] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Bootstrap starting.
I20260807 06:49:24.314330 7458 raft_consensus.cc:2229] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:49:24.314556 7455 ts_tablet_manager.cc:1905] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:24.315073 7458 raft_consensus.cc:2258] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:24.316560 7422 tablet_bootstrap.cc:654] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.319295 7454 tablet_replica.cc:330] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: stopping tablet replica
I20260807 06:49:24.319833 7454 raft_consensus.cc:2229] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:24.320390 7454 raft_consensus.cc:2258] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:24.321414 7421 tablet_bootstrap.cc:492] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: No bootstrap required, opened a new log
I20260807 06:49:24.321637 7421 ts_tablet_manager.cc:1397] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20260807 06:49:24.322957 7422 tablet_bootstrap.cc:492] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: No bootstrap required, opened a new log
I20260807 06:49:24.323211 7422 ts_tablet_manager.cc:1397] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260807 06:49:24.324029 7421 raft_consensus.cc:348] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.324375 7421 raft_consensus.cc:374] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.324411 7454 ts_tablet_manager.cc:1905] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:24.324591 7421 raft_consensus.cc:729] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.324487 7422 raft_consensus.cc:348] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.324703 7422 raft_consensus.cc:374] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.324819 7422 raft_consensus.cc:729] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc7b396e5814956a0b6034c6b45e158, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.325076 7421 consensus_queue.cc:260] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.325170 7422 consensus_queue.cc:260] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.326540 7422 ts_tablet_manager.cc:1428] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 06:49:24.326877 7458 ts_tablet_manager.cc:1905] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:24.331852 7454 ts_tablet_manager.cc:1918] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:24.332024 7420 tablet_bootstrap.cc:492] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: No bootstrap required, opened a new log
I20260807 06:49:24.332096 7454 log.cc:1196] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/44f7c030991a4f9e97473ea34d9287bf
I20260807 06:49:24.332239 7420 ts_tablet_manager.cc:1397] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20260807 06:49:24.332722 7454 ts_tablet_manager.cc:1939] T 44f7c030991a4f9e97473ea34d9287bf P d0fb4917e8d64999bcc270395ba1f9b2: Deleting consensus metadata
I20260807 06:49:24.333173 7421 ts_tablet_manager.cc:1428] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20260807 06:49:24.333210 7420 raft_consensus.cc:348] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.333463 7420 raft_consensus.cc:374] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.333567 7420 raft_consensus.cc:729] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.333837 7420 consensus_queue.cc:260] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.334481 7420 ts_tablet_manager.cc:1428] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 06:49:24.334810 6613 catalog_manager.cc:4841] TS d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713): tablet 44f7c030991a4f9e97473ea34d9287bf (table test-table [id=05372b68a02a4de58c88a1af68b5290b]) successfully deleted
I20260807 06:49:24.335868 7458 ts_tablet_manager.cc:1918] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:24.336186 7458 log.cc:1196] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/44f7c030991a4f9e97473ea34d9287bf
I20260807 06:49:24.337008 7458 ts_tablet_manager.cc:1939] T 44f7c030991a4f9e97473ea34d9287bf P bfaefa9bf4394f328ea027a2c8787975: Deleting consensus metadata
I20260807 06:49:24.338907 6614 catalog_manager.cc:4841] TS bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149): tablet 44f7c030991a4f9e97473ea34d9287bf (table test-table [id=05372b68a02a4de58c88a1af68b5290b]) successfully deleted
I20260807 06:49:24.342411 7426 raft_consensus.cc:482] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:24.342695 7426 raft_consensus.cc:504] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.343643 7455 ts_tablet_manager.cc:1918] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:24.343842 7455 log.cc:1196] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/44f7c030991a4f9e97473ea34d9287bf
I20260807 06:49:24.344430 7455 ts_tablet_manager.cc:1939] T 44f7c030991a4f9e97473ea34d9287bf P 8dc7b396e5814956a0b6034c6b45e158: Deleting consensus metadata
I20260807 06:49:24.344412 7426 leader_election.cc:288] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063), bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:24.346408 6611 catalog_manager.cc:4841] TS 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063): tablet 44f7c030991a4f9e97473ea34d9287bf (table test-table [id=05372b68a02a4de58c88a1af68b5290b]) successfully deleted
I20260807 06:49:24.353766 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.345008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:40235 (local address 127.27.54.66:33063)
0807 06:49:24.345175 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.345184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.345254 (+ 70us) server_negotiation.cc:407] Connection header received
0807 06:49:24.345554 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.345559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.345649 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.345726 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:24.346696 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.348231 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.351471 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.352641 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.352756 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.352809 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.353321 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.353459 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:49:24.353826 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.344599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:33063 (local address 127.27.54.65:40235)
0807 06:49:24.345136 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.345158 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.345367 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.345978 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.345983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.346012 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:24.346437 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.346450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.348499 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.348508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.351270 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.351288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.352996 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.353001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.353146 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.353168 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.353293 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.353445 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":1693,"threads_started":1}
I20260807 06:49:24.354790 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6f6c5810a43709af6ac5a7d0ed7ca" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158" is_pre_election: true
I20260807 06:49:24.355072 7077 raft_consensus.cc:2454] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 0.
I20260807 06:49:24.355686 6765 leader_election.cc:302] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc7b396e5814956a0b6034c6b45e158, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:24.356171 7426 raft_consensus.cc:2781] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:24.356304 7426 raft_consensus.cc:482] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:24.356413 7426 raft_consensus.cc:3037] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.357636 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.345939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:60053 (local address 127.27.54.67:33149)
0807 06:49:24.346127 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.346136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.350150 (+ 4014us) server_negotiation.cc:407] Connection header received
0807 06:49:24.350330 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.350335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.350416 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.350517 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:49:24.351308 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.352543 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.355222 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.356419 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.356629 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.356686 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.357173 (+ 487us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.357315 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:49:24.357755 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.345553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:33149 (local address 127.27.54.65:60053)
0807 06:49:24.350012 (+ 4459us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.350036 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.350186 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.350552 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.350557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.350585 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:49:24.351063 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.351079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.352760 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.352765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.354969 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.354985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.356836 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.356841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.356976 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.357000 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.357115 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.357265 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4140,"mutex_wait_us":1,"thread_start_us":3921,"threads_started":1}
I20260807 06:49:24.358853 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6f6c5810a43709af6ac5a7d0ed7ca" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaefa9bf4394f328ea027a2c8787975" is_pre_election: true
I20260807 06:49:24.359370 7300 raft_consensus.cc:2454] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 0.
I20260807 06:49:24.360256 7426 raft_consensus.cc:504] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.361079 7426 leader_election.cc:288] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063), bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:24.361652 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6f6c5810a43709af6ac5a7d0ed7ca" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158"
I20260807 06:49:24.361899 7077 raft_consensus.cc:3037] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.363016 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6f6c5810a43709af6ac5a7d0ed7ca" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaefa9bf4394f328ea027a2c8787975"
I20260807 06:49:24.363245 7300 raft_consensus.cc:3037] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.365166 7077 raft_consensus.cc:2454] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 1.
I20260807 06:49:24.366112 7300 raft_consensus.cc:2454] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 1.
I20260807 06:49:24.366722 6771 leader_election.cc:302] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:24.371035 7426 raft_consensus.cc:2781] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:24.371654 7426 raft_consensus.cc:686] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 LEADER]: Becoming Leader. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Running, Role: LEADER
I20260807 06:49:24.372161 7426 consensus_queue.cc:237] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.376216 6628 catalog_manager.cc:5495] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65). New cstate: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } health_report { overall_health: UNKNOWN } } }
Aug 07 06:49:24 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085360, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.126@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:24.433379 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.417796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41245 (local address 127.0.0.1:53250)
0807 06:49:24.418591 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.418626 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.418872 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.419304 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.419311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.420077 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:24.420301 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.420312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.421525 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.421532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.422739 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.422753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.423534 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.423540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.423740 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.424507 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:24.424526 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:24.427118 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:24.430538 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:24.430552 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:24.430561 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:24.431076 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:24.431456 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:24.431462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:24.431465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:24.431653 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:24.432201 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:24.432209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:24.432403 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.432569 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.432924 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":443,"threads_started":1}
I20260807 06:49:24.433410 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.417847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.27.54.126:41245)
0807 06:49:24.419008 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.419013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.419034 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:49:24.419088 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.419091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.419148 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.419265 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:24.420480 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.421370 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.422953 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.423355 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.423509 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.427357 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:24.427362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:24.427366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:24.427394 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:24.430397 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:24.431209 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:24.431219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:24.431221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:24.431337 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:24.431774 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:24.431783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:24.431785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:24.432067 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:24.432226 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.432683 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.433052 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":985,"thread_start_us":723,"threads_started":1}
I20260807 06:49:24.439229 6628 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:53250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 06:49:24.440302 6628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:24.447868 6837 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc27bf6a8b64f9d9875c152b9547600 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bde131a77b242fe84bcf928ff5eaa94]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:49:24.447965 7043 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc27bf6a8b64f9d9875c152b9547600 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bde131a77b242fe84bcf928ff5eaa94]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:49:24.448205 7271 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc27bf6a8b64f9d9875c152b9547600 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bde131a77b242fe84bcf928ff5eaa94]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 06:49:24.448483 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc27bf6a8b64f9d9875c152b9547600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.448537 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc27bf6a8b64f9d9875c152b9547600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.448807 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc27bf6a8b64f9d9875c152b9547600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.456547 7420 tablet_bootstrap.cc:492] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: Bootstrap starting.
I20260807 06:49:24.456766 7421 tablet_bootstrap.cc:492] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: Bootstrap starting.
I20260807 06:49:24.458523 7420 tablet_bootstrap.cc:654] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.458545 7421 tablet_bootstrap.cc:654] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.459816 7422 tablet_bootstrap.cc:492] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: Bootstrap starting.
I20260807 06:49:24.461459 7420 tablet_bootstrap.cc:492] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: No bootstrap required, opened a new log
I20260807 06:49:24.461642 7420 ts_tablet_manager.cc:1397] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 06:49:24.461692 7422 tablet_bootstrap.cc:654] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.462456 7420 raft_consensus.cc:348] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.462771 7420 raft_consensus.cc:374] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.462885 7420 raft_consensus.cc:729] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.463199 7420 consensus_queue.cc:260] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.463641 7426 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.463980 7426 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.464192 7421 tablet_bootstrap.cc:492] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: No bootstrap required, opened a new log
I20260807 06:49:24.464377 7421 ts_tablet_manager.cc:1397] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260807 06:49:24.465065 7420 ts_tablet_manager.cc:1428] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:24.466362 7426 raft_consensus.cc:482] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:24.466349 7421 raft_consensus.cc:348] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.466609 7421 raft_consensus.cc:374] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.466521 7426 raft_consensus.cc:504] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.466719 7421 raft_consensus.cc:729] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.467023 7421 consensus_queue.cc:260] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.467193 7426 leader_election.cc:288] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063), bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:24.467440 7432 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.467816 7432 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.467845 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc27bf6a8b64f9d9875c152b9547600" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaefa9bf4394f328ea027a2c8787975" is_pre_election: true
I20260807 06:49:24.468115 7300 raft_consensus.cc:2454] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 0.
I20260807 06:49:24.468356 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc27bf6a8b64f9d9875c152b9547600" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158" is_pre_election: true
I20260807 06:49:24.468544 6771 leader_election.cc:302] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:24.468842 7426 raft_consensus.cc:2781] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:24.468953 7426 raft_consensus.cc:482] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:24.469075 7426 raft_consensus.cc:3037] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Advancing to term 1
W20260807 06:49:24.469302 6765 leader_election.cc:341] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063): Illegal state: must be running to vote when last-logged opid is not known
I20260807 06:49:24.471724 7426 raft_consensus.cc:504] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.472200 7421 ts_tablet_manager.cc:1428] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260807 06:49:24.472966 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc27bf6a8b64f9d9875c152b9547600" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158"
I20260807 06:49:24.473457 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc27bf6a8b64f9d9875c152b9547600" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaefa9bf4394f328ea027a2c8787975"
W20260807 06:49:24.473536 6765 leader_election.cc:341] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063): Illegal state: must be running to vote when last-logged opid is not known
I20260807 06:49:24.473707 7300 raft_consensus.cc:3037] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.475050 7422 tablet_bootstrap.cc:492] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: No bootstrap required, opened a new log
I20260807 06:49:24.475229 7422 ts_tablet_manager.cc:1397] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20260807 06:49:24.475996 7422 raft_consensus.cc:348] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.476244 7422 raft_consensus.cc:374] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.476331 7422 raft_consensus.cc:729] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc7b396e5814956a0b6034c6b45e158, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.476471 7300 raft_consensus.cc:2454] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 1.
I20260807 06:49:24.476775 7426 leader_election.cc:288] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063), bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:24.476956 6771 leader_election.cc:302] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters: 8dc7b396e5814956a0b6034c6b45e158
I20260807 06:49:24.477265 7426 raft_consensus.cc:2781] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:24.477197 7422 consensus_queue.cc:260] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.477440 7426 raft_consensus.cc:686] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 LEADER]: Becoming Leader. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Running, Role: LEADER
I20260807 06:49:24.477676 7428 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.478044 7428 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.477921 7426 consensus_queue.cc:237] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.479031 7422 ts_tablet_manager.cc:1428] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 06:49:24.481911 7468 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: TxnStatusTablet state changed. Reason: New leader d0fb4917e8d64999bcc270395ba1f9b2. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.482297 7468 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:49:24.484262 6628 catalog_manager.cc:5495] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65). New cstate: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } health_report { overall_health: UNKNOWN } } }
I20260807 06:49:24.484575 7485 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 06:49:24.484825 7485 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 06:49:24.510228 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.502623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.27.54.65:44713)
0807 06:49:24.502838 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.502846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.503012 (+ 166us) server_negotiation.cc:407] Connection header received
0807 06:49:24.503261 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.503269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.503381 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.503546 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:24.503978 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.505203 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.507254 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.507962 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.508109 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.508726 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:24.509472 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:24.509631 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.509761 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.509866 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:49:24.510253 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.502620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44713 (local address 127.0.0.1:51634)
0807 06:49:24.502898 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.502920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.503066 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.503588 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.503593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.503607 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:49:24.503824 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.503836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.505377 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.505383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.507066 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.507083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.508156 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.508162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.508414 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.508502 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:49:24.509609 (+ 1107us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:49:24.509637 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.509735 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.509910 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 06:49:24.515589 7077 raft_consensus.cc:3037] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:24.515987 7300 raft_consensus.cc:1261] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Refusing update from remote peer d0fb4917e8d64999bcc270395ba1f9b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:24.517066 7426 consensus_queue.cc:1035] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:24.519273 7077 raft_consensus.cc:1261] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Refusing update from remote peer d0fb4917e8d64999bcc270395ba1f9b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:24.520002 7468 consensus_queue.cc:1035] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:24.529143 7432 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: TxnStatusTablet state changed. Reason: New leader d0fb4917e8d64999bcc270395ba1f9b2. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.529556 7432 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.534150 7428 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: TxnStatusTablet state changed. Reason: New leader d0fb4917e8d64999bcc270395ba1f9b2. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.534482 7428 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.534819 7468 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.535085 7468 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:49:24.541281 7432 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.541620 7432 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P bfaefa9bf4394f328ea027a2c8787975: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.545450 7426 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.545799 7426 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P d0fb4917e8d64999bcc270395ba1f9b2: This TxnStatusTablet replica's current role is: LEADER
I20260807 06:49:24.550588 7428 tablet_replica.cc:439] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } } }
I20260807 06:49:24.550906 7428 tablet_replica.cc:442] T 3cc27bf6a8b64f9d9875c152b9547600 P 8dc7b396e5814956a0b6034c6b45e158: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 06:49:24.553138 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.543678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.27.54.65:44713)
0807 06:49:24.543844 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.543864 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.543885 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:49:24.543955 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.543960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.544062 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.544162 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:24.546064 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.547335 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.549505 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.550241 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.550377 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.551898 (+ 1521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:24.552035 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:24.552121 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.552707 (+ 586us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.552796 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:49:24.573915 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.565921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:57499 (local address 127.27.54.65:44713)
0807 06:49:24.566188 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.566196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.566880 (+ 684us) server_negotiation.cc:407] Connection header received
0807 06:49:24.567105 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.567111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.567204 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.567316 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:24.568017 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.569210 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.571510 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.572247 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.572388 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.572965 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:24.573131 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:24.573237 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.573435 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.573521 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":80}
I20260807 06:49:24.574308 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.565950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44713 (local address 127.27.54.65:57499)
0807 06:49:24.566753 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:24.566774 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:24.566950 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:24.567357 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:24.567362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:24.567376 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:49:24.567822 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.567836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.569360 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.569366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.571343 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:24.571360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.572481 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.572488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:24.572685 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.572765 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:49:24.573275 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:49:24.573288 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 06:49:24.573395 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 06:49:24.573563 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":413,"threads_started":1}
I20260807 06:49:24.593482 7300 raft_consensus.cc:1261] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Refusing update from remote peer d0fb4917e8d64999bcc270395ba1f9b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:24.594794 7468 consensus_queue.cc:1035] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:24.594614 7077 raft_consensus.cc:1261] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Refusing update from remote peer d0fb4917e8d64999bcc270395ba1f9b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:24.598403 7468 consensus_queue.cc:1035] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:24.610994 7443 mvcc.cc:205] Tried to move back new op lower bound from 7315805653364367360 to 7315805652472844288. Current Snapshot: MvccSnapshot[applied={T|T < 7315805653364367360}]
I20260807 06:49:24.727221 6628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:24.727665 6628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:24.731864 6628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Sending DeleteTablet for 3 replicas of tablet 87c6f6c5810a43709af6ac5a7d0ed7ca
I20260807 06:49:24.732861 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:24 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:36886
I20260807 06:49:24.732928 6837 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:24 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:51630
I20260807 06:49:24.733301 7455 tablet_replica.cc:330] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: stopping tablet replica
I20260807 06:49:24.733531 7455 raft_consensus.cc:2229] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:24.733757 7455 raft_consensus.cc:2258] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:24.733995 7271 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c6f6c5810a43709af6ac5a7d0ed7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:24 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47672
I20260807 06:49:24.734086 7454 tablet_replica.cc:330] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: stopping tablet replica
I20260807 06:49:24.734304 7454 raft_consensus.cc:2229] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:49:24.734565 7458 tablet_replica.cc:330] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: stopping tablet replica
I20260807 06:49:24.734794 7458 raft_consensus.cc:2229] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:24.734964 7454 raft_consensus.cc:2258] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:24.735028 7458 raft_consensus.cc:2258] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:24.736279 7455 ts_tablet_manager.cc:1905] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:24.736354 7458 ts_tablet_manager.cc:1905] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:24.736379 7454 ts_tablet_manager.cc:1905] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:24.743966 7454 ts_tablet_manager.cc:1918] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:49:24.743985 7458 ts_tablet_manager.cc:1918] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:49:24.744139 7454 log.cc:1196] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/87c6f6c5810a43709af6ac5a7d0ed7ca
I20260807 06:49:24.744167 7458 log.cc:1196] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/87c6f6c5810a43709af6ac5a7d0ed7ca
I20260807 06:49:24.744627 7454 ts_tablet_manager.cc:1939] T 87c6f6c5810a43709af6ac5a7d0ed7ca P d0fb4917e8d64999bcc270395ba1f9b2: Deleting consensus metadata
I20260807 06:49:24.744649 7458 ts_tablet_manager.cc:1939] T 87c6f6c5810a43709af6ac5a7d0ed7ca P bfaefa9bf4394f328ea027a2c8787975: Deleting consensus metadata
I20260807 06:49:24.745864 6613 catalog_manager.cc:4841] TS d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713): tablet 87c6f6c5810a43709af6ac5a7d0ed7ca (table test-table [id=37702ed62d474be4a10514326fa7378f]) successfully deleted
I20260807 06:49:24.745862 6614 catalog_manager.cc:4841] TS bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149): tablet 87c6f6c5810a43709af6ac5a7d0ed7ca (table test-table [id=37702ed62d474be4a10514326fa7378f]) successfully deleted
I20260807 06:49:24.751118 7455 ts_tablet_manager.cc:1918] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:49:24.751389 7455 log.cc:1196] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/87c6f6c5810a43709af6ac5a7d0ed7ca
I20260807 06:49:24.751963 7455 ts_tablet_manager.cc:1939] T 87c6f6c5810a43709af6ac5a7d0ed7ca P 8dc7b396e5814956a0b6034c6b45e158: Deleting consensus metadata
I20260807 06:49:24.753333 6611 catalog_manager.cc:4841] TS 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063): tablet 87c6f6c5810a43709af6ac5a7d0ed7ca (table test-table [id=37702ed62d474be4a10514326fa7378f]) successfully deleted
Aug 07 06:49:24 dist-test-slave-3kk6 krb5kdc[6570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 06:49:24 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:24 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:49:24.833742 7476 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:24.819120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.27.54.126:41245)
0807 06:49:24.819341 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.819348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.819902 (+ 554us) server_negotiation.cc:407] Connection header received
0807 06:49:24.820092 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.820098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.820187 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.820311 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:24.822090 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.823349 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.825318 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.826044 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.826191 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.833479 (+ 7288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53258: BlockingRecv error: recv got EOF from 127.0.0.1:53258 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 06:49:24 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM
I20260807 06:49:24.856658 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:24.837625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.27.54.126:41245)
0807 06:49:24.837784 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:24.837791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:24.838302 (+ 511us) server_negotiation.cc:407] Connection header received
0807 06:49:24.838504 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:24.838509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:24.838591 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:24.838734 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:24.840415 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.841557 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.843617 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:24.844291 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:24.844431 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:24.849648 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:24.849655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:24.849660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:24.849717 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:24.853348 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:24.854175 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:24.854180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:24.854183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:24.854373 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:24.854995 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:24.855000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:24.855003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:24.855391 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:24.855705 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:24.856020 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 06:49:24.856283 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 06:49:24.861403 6628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:49:24.862048 6628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:24.869410 6837 tablet_service.cc:1467] Processing CreateTablet for tablet 57c2b0ec34d3468ba93ba3004e302062 (DEFAULT_TABLE table=test-table [id=7cf638f008794a7fb53e39a244c00bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:24.869972 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c2b0ec34d3468ba93ba3004e302062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.869868 7043 tablet_service.cc:1467] Processing CreateTablet for tablet 57c2b0ec34d3468ba93ba3004e302062 (DEFAULT_TABLE table=test-table [id=7cf638f008794a7fb53e39a244c00bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:24.870128 7271 tablet_service.cc:1467] Processing CreateTablet for tablet 57c2b0ec34d3468ba93ba3004e302062 (DEFAULT_TABLE table=test-table [id=7cf638f008794a7fb53e39a244c00bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:24.870337 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c2b0ec34d3468ba93ba3004e302062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.870553 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c2b0ec34d3468ba93ba3004e302062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:24.877090 7422 tablet_bootstrap.cc:492] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Bootstrap starting.
I20260807 06:49:24.877089 7421 tablet_bootstrap.cc:492] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Bootstrap starting.
I20260807 06:49:24.878895 7421 tablet_bootstrap.cc:654] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.878895 7422 tablet_bootstrap.cc:654] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.879773 7420 tablet_bootstrap.cc:492] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Bootstrap starting.
I20260807 06:49:24.881641 7420 tablet_bootstrap.cc:654] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:24.888914 7422 tablet_bootstrap.cc:492] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: No bootstrap required, opened a new log
I20260807 06:49:24.888959 7420 tablet_bootstrap.cc:492] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: No bootstrap required, opened a new log
I20260807 06:49:24.889111 7420 ts_tablet_manager.cc:1397] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260807 06:49:24.889104 7422 ts_tablet_manager.cc:1397] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260807 06:49:24.889979 7420 raft_consensus.cc:348] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.890340 7420 raft_consensus.cc:374] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.890488 7420 raft_consensus.cc:729] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.890884 7421 tablet_bootstrap.cc:492] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: No bootstrap required, opened a new log
I20260807 06:49:24.890723 7422 raft_consensus.cc:348] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.890787 7420 consensus_queue.cc:260] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.891019 7422 raft_consensus.cc:374] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.891122 7422 raft_consensus.cc:729] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc7b396e5814956a0b6034c6b45e158, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.891287 7421 ts_tablet_manager.cc:1397] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260807 06:49:24.891423 7420 ts_tablet_manager.cc:1428] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 06:49:24.891396 7422 consensus_queue.cc:260] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.891975 7422 ts_tablet_manager.cc:1428] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260807 06:49:24.892429 7421 raft_consensus.cc:348] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.892781 7421 raft_consensus.cc:374] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:24.892928 7421 raft_consensus.cc:729] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Initialized, Role: FOLLOWER
I20260807 06:49:24.893250 7421 consensus_queue.cc:260] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:24.896118 7421 ts_tablet_manager.cc:1428] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260807 06:49:25.049782 7426 raft_consensus.cc:482] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:25.050068 7426 raft_consensus.cc:504] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.050949 7426 leader_election.cc:288] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063), bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:25.051960 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c2b0ec34d3468ba93ba3004e302062" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158" is_pre_election: true
I20260807 06:49:25.052237 7077 raft_consensus.cc:2454] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 0.
I20260807 06:49:25.052193 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c2b0ec34d3468ba93ba3004e302062" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaefa9bf4394f328ea027a2c8787975" is_pre_election: true
I20260807 06:49:25.052599 7300 raft_consensus.cc:2454] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 0.
I20260807 06:49:25.052834 6765 leader_election.cc:302] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc7b396e5814956a0b6034c6b45e158, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:25.053167 7426 raft_consensus.cc:2781] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:25.053301 7426 raft_consensus.cc:482] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:25.053426 7426 raft_consensus.cc:3037] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:25.055778 7426 raft_consensus.cc:504] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.056442 7426 leader_election.cc:288] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063), bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149)
I20260807 06:49:25.056964 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c2b0ec34d3468ba93ba3004e302062" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaefa9bf4394f328ea027a2c8787975"
I20260807 06:49:25.056974 7077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c2b0ec34d3468ba93ba3004e302062" candidate_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158"
I20260807 06:49:25.057191 7077 raft_consensus.cc:3037] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:25.057184 7300 raft_consensus.cc:3037] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:25.059387 7300 raft_consensus.cc:2454] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 1.
I20260807 06:49:25.059823 6771 leader_election.cc:302] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:25.060117 7426 raft_consensus.cc:2781] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:25.060335 7426 raft_consensus.cc:686] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 LEADER]: Becoming Leader. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Running, Role: LEADER
I20260807 06:49:25.060302 7077 raft_consensus.cc:2454] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb4917e8d64999bcc270395ba1f9b2 in term 1.
I20260807 06:49:25.060611 7426 consensus_queue.cc:237] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.063925 6632 catalog_manager.cc:5495] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65). New cstate: current_term: 1 leader_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } health_report { overall_health: UNKNOWN } } }
I20260807 06:49:25.087106 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.080730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.27.54.65:44713)
0807 06:49:25.081481 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.081486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.081519 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:49:25.081575 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.081579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.081651 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.081730 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:25.082225 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.083119 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.085227 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.085668 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.085764 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.086287 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:25.086399 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:25.086493 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.086690 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.086772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":461,"threads_started":1}
I20260807 06:49:25.090394 7077 raft_consensus.cc:1261] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Refusing update from remote peer d0fb4917e8d64999bcc270395ba1f9b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:25.090583 7300 raft_consensus.cc:1261] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Refusing update from remote peer d0fb4917e8d64999bcc270395ba1f9b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:25.091285 7426 consensus_queue.cc:1035] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:25.091645 7498 consensus_queue.cc:1035] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:25.112752 6632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:25.113030 6632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:25.116683 6632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Sending DeleteTablet for 3 replicas of tablet 57c2b0ec34d3468ba93ba3004e302062
I20260807 06:49:25.117549 6837 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c2b0ec34d3468ba93ba3004e302062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:25 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:51630
I20260807 06:49:25.118384 7454 tablet_replica.cc:330] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: stopping tablet replica
I20260807 06:49:25.118625 7454 raft_consensus.cc:2229] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:49:25.118763 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c2b0ec34d3468ba93ba3004e302062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:25 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:36886
I20260807 06:49:25.119037 7454 raft_consensus.cc:2258] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:25.119119 7455 tablet_replica.cc:330] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: stopping tablet replica
I20260807 06:49:25.119304 7455 raft_consensus.cc:2229] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:25.119540 7455 raft_consensus.cc:2258] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:25.119947 7271 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c2b0ec34d3468ba93ba3004e302062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:25 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47672
I20260807 06:49:25.120257 7458 tablet_replica.cc:330] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: stopping tablet replica
I20260807 06:49:25.120427 7458 raft_consensus.cc:2229] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:25.120671 7455 ts_tablet_manager.cc:1905] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:25.120703 7458 raft_consensus.cc:2258] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:25.121712 7454 ts_tablet_manager.cc:1905] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:25.122805 7458 ts_tablet_manager.cc:1905] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:25.127501 6632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:49:25.128600 7455 ts_tablet_manager.cc:1918] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:25.128827 7455 log.cc:1196] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/57c2b0ec34d3468ba93ba3004e302062
W20260807 06:49:25.129050 6632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:49:25.129371 7458 ts_tablet_manager.cc:1918] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:25.129519 7458 log.cc:1196] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/57c2b0ec34d3468ba93ba3004e302062
I20260807 06:49:25.129544 7455 ts_tablet_manager.cc:1939] T 57c2b0ec34d3468ba93ba3004e302062 P 8dc7b396e5814956a0b6034c6b45e158: Deleting consensus metadata
I20260807 06:49:25.129981 7458 ts_tablet_manager.cc:1939] T 57c2b0ec34d3468ba93ba3004e302062 P bfaefa9bf4394f328ea027a2c8787975: Deleting consensus metadata
I20260807 06:49:25.131542 6614 catalog_manager.cc:4841] TS bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149): tablet 57c2b0ec34d3468ba93ba3004e302062 (table test-table [id=7cf638f008794a7fb53e39a244c00bba]) successfully deleted
I20260807 06:49:25.131788 6611 catalog_manager.cc:4841] TS 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063): tablet 57c2b0ec34d3468ba93ba3004e302062 (table test-table [id=7cf638f008794a7fb53e39a244c00bba]) successfully deleted
I20260807 06:49:25.132983 7454 ts_tablet_manager.cc:1918] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:49:25.133394 7454 log.cc:1196] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/57c2b0ec34d3468ba93ba3004e302062
I20260807 06:49:25.133834 7454 ts_tablet_manager.cc:1939] T 57c2b0ec34d3468ba93ba3004e302062 P d0fb4917e8d64999bcc270395ba1f9b2: Deleting consensus metadata
I20260807 06:49:25.135258 6613 catalog_manager.cc:4841] TS d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713): tablet 57c2b0ec34d3468ba93ba3004e302062 (table test-table [id=7cf638f008794a7fb53e39a244c00bba]) successfully deleted
I20260807 06:49:25.140094 7043 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0ea3c009724a59aeed87a9149a3168 (DEFAULT_TABLE table=test-table [id=7a4b85420cd74f94bed7aac5b3c07e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:25.140151 6837 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0ea3c009724a59aeed87a9149a3168 (DEFAULT_TABLE table=test-table [id=7a4b85420cd74f94bed7aac5b3c07e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:25.140470 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0ea3c009724a59aeed87a9149a3168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:25.140578 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0ea3c009724a59aeed87a9149a3168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:25.145187 7422 tablet_bootstrap.cc:492] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Bootstrap starting.
I20260807 06:49:25.146759 7422 tablet_bootstrap.cc:654] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:25.147204 7271 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0ea3c009724a59aeed87a9149a3168 (DEFAULT_TABLE table=test-table [id=7a4b85420cd74f94bed7aac5b3c07e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:49:25.147680 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0ea3c009724a59aeed87a9149a3168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:25.147863 7420 tablet_bootstrap.cc:492] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Bootstrap starting.
I20260807 06:49:25.149706 7420 tablet_bootstrap.cc:654] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:25.155335 7421 tablet_bootstrap.cc:492] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Bootstrap starting.
I20260807 06:49:25.156180 7420 tablet_bootstrap.cc:492] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: No bootstrap required, opened a new log
I20260807 06:49:25.156394 7420 ts_tablet_manager.cc:1397] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260807 06:49:25.157071 7421 tablet_bootstrap.cc:654] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:25.157254 7420 raft_consensus.cc:348] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.157598 7420 raft_consensus.cc:374] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:25.157706 7420 raft_consensus.cc:729] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb4917e8d64999bcc270395ba1f9b2, State: Initialized, Role: FOLLOWER
I20260807 06:49:25.158241 7420 consensus_queue.cc:260] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.159298 7422 tablet_bootstrap.cc:492] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: No bootstrap required, opened a new log
I20260807 06:49:25.159533 7422 ts_tablet_manager.cc:1397] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20260807 06:49:25.160786 7422 raft_consensus.cc:348] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.161130 7422 raft_consensus.cc:374] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:25.161263 7422 raft_consensus.cc:729] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc7b396e5814956a0b6034c6b45e158, State: Initialized, Role: FOLLOWER
I20260807 06:49:25.161581 7422 consensus_queue.cc:260] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.163143 7422 ts_tablet_manager.cc:1428] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260807 06:49:25.163818 7420 ts_tablet_manager.cc:1428] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260807 06:49:25.165493 7421 tablet_bootstrap.cc:492] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: No bootstrap required, opened a new log
I20260807 06:49:25.165689 7421 ts_tablet_manager.cc:1397] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260807 06:49:25.166524 7421 raft_consensus.cc:348] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.166874 7421 raft_consensus.cc:374] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:25.167150 7421 raft_consensus.cc:729] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Initialized, Role: FOLLOWER
I20260807 06:49:25.167429 7421 consensus_queue.cc:260] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.168035 7421 ts_tablet_manager.cc:1428] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 06:49:25.258939 7432 raft_consensus.cc:482] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:49:25.259194 7432 raft_consensus.cc:504] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.259959 7432 leader_election.cc:288] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713), 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063)
I20260807 06:49:25.260555 7074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0ea3c009724a59aeed87a9149a3168" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158" is_pre_election: true
I20260807 06:49:25.260576 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0ea3c009724a59aeed87a9149a3168" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" is_pre_election: true
I20260807 06:49:25.260819 7074 raft_consensus.cc:2454] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 0.
I20260807 06:49:25.260885 6870 raft_consensus.cc:2454] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 0.
I20260807 06:49:25.261356 7186 leader_election.cc:302] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:25.261698 7432 raft_consensus.cc:2781] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:49:25.261811 7432 raft_consensus.cc:482] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:49:25.261950 7432 raft_consensus.cc:3037] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:25.264222 7432 raft_consensus.cc:504] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.264824 7432 leader_election.cc:288] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 election: Requested vote from peers d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713), 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063)
I20260807 06:49:25.265380 6870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0ea3c009724a59aeed87a9149a3168" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb4917e8d64999bcc270395ba1f9b2"
I20260807 06:49:25.265385 7074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0ea3c009724a59aeed87a9149a3168" candidate_uuid: "bfaefa9bf4394f328ea027a2c8787975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc7b396e5814956a0b6034c6b45e158"
I20260807 06:49:25.265594 7074 raft_consensus.cc:3037] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:25.265612 6870 raft_consensus.cc:3037] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:25.267920 6870 raft_consensus.cc:2454] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 1.
I20260807 06:49:25.268374 7186 leader_election.cc:302] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfaefa9bf4394f328ea027a2c8787975, d0fb4917e8d64999bcc270395ba1f9b2; no voters:
I20260807 06:49:25.268658 7432 raft_consensus.cc:2781] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:25.268672 7074 raft_consensus.cc:2454] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaefa9bf4394f328ea027a2c8787975 in term 1.
I20260807 06:49:25.270902 7432 raft_consensus.cc:686] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 1 LEADER]: Becoming Leader. State: Replica: bfaefa9bf4394f328ea027a2c8787975, State: Running, Role: LEADER
I20260807 06:49:25.271399 7432 consensus_queue.cc:237] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } }
I20260807 06:49:25.274240 6632 catalog_manager.cc:5495] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67). New cstate: current_term: 1 leader_uuid: "bfaefa9bf4394f328ea027a2c8787975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaefa9bf4394f328ea027a2c8787975" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33149 } health_report { overall_health: HEALTHY } } }
I20260807 06:49:25.299340 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.293410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.27.54.67:33149)
0807 06:49:25.294016 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.294020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.294035 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:49:25.294088 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.294091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.294150 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.294232 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:25.294744 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.295504 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.297493 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.297905 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.297996 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.298510 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:25.298608 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:25.298768 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.298936 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.299001 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":368,"threads_started":1}
I20260807 06:49:25.315081 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.309166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:41739 (local address 127.27.54.65:44713)
0807 06:49:25.309333 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.309347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.309902 (+ 555us) server_negotiation.cc:407] Connection header received
0807 06:49:25.310137 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.310141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.310209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.310318 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:25.310874 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.311712 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.313241 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.313699 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.313860 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.314278 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:25.314383 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:25.314478 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.314630 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.314750 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:49:25.315172 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.309142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44713 (local address 127.27.54.67:41739)
0807 06:49:25.309788 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:25.309805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:25.309980 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:25.310333 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:25.310346 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:25.310358 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:49:25.310637 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:25.310686 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.311863 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.311871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:25.313096 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:25.313108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.313885 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.313892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:25.314030 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.314088 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:49:25.314497 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:49:25.314511 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 06:49:25.314607 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 06:49:25.314775 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":338,"threads_started":1}
I20260807 06:49:25.324571 6870 raft_consensus.cc:1261] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Refusing update from remote peer bfaefa9bf4394f328ea027a2c8787975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:25.324628 7074 raft_consensus.cc:1261] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Refusing update from remote peer bfaefa9bf4394f328ea027a2c8787975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:49:25.325294 7432 consensus_queue.cc:1035] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb4917e8d64999bcc270395ba1f9b2" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:25.325569 7534 consensus_queue.cc:1035] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc7b396e5814956a0b6034c6b45e158" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:49:25.365114 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.356046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:46427 (local address 127.27.54.67:33149)
0807 06:49:25.356260 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.356268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.359122 (+ 2854us) server_negotiation.cc:407] Connection header received
0807 06:49:25.359372 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.359381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.359477 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.359579 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:49:25.360194 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.361341 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.363204 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.363864 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.364008 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.364373 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:49:25.364508 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:49:25.364610 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.364732 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.364824 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:49:25.365191 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.355944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:33149 (local address 127.27.54.65:46427)
0807 06:49:25.359002 (+ 3058us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:25.359019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:25.359183 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:25.359613 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:25.359617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:25.359630 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 06:49:25.359986 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:25.359995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.361504 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.361510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:25.363041 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:25.363054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.364031 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.364036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:25.364166 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.364205 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 06:49:25.364625 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 06:49:25.364633 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:49:25.364684 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 06:49:25.364808 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2910,"thread_start_us":2734,"threads_started":1}
I20260807 06:49:25.411605 6628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:25.411885 6628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:49:25.415012 6628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4575be679374e2b90b977553dc6722c: Sending DeleteTablet for 3 replicas of tablet 4a0ea3c009724a59aeed87a9149a3168
I20260807 06:49:25.416316 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0ea3c009724a59aeed87a9149a3168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:25 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:36886
I20260807 06:49:25.416667 7455 tablet_replica.cc:330] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: stopping tablet replica
I20260807 06:49:25.416895 7455 raft_consensus.cc:2229] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:25.417137 7455 raft_consensus.cc:2258] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:25.417640 7271 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0ea3c009724a59aeed87a9149a3168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:25 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47672
I20260807 06:49:25.417935 7458 tablet_replica.cc:330] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: stopping tablet replica
I20260807 06:49:25.418159 7458 raft_consensus.cc:2229] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:49:25.418475 7458 raft_consensus.cc:2258] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:25.419456 7458 ts_tablet_manager.cc:1905] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:25.423280 6837 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0ea3c009724a59aeed87a9149a3168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:49:25 UTC) from {username='oryx', principal='oryx/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:51630
I20260807 06:49:25.423593 7455 ts_tablet_manager.cc:1905] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:25.426486 7458 ts_tablet_manager.cc:1918] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:49:25.426627 7458 log.cc:1196] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/4a0ea3c009724a59aeed87a9149a3168
I20260807 06:49:25.426874 7454 tablet_replica.cc:330] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: stopping tablet replica
I20260807 06:49:25.427110 7458 ts_tablet_manager.cc:1939] T 4a0ea3c009724a59aeed87a9149a3168 P bfaefa9bf4394f328ea027a2c8787975: Deleting consensus metadata
I20260807 06:49:25.427121 7454 raft_consensus.cc:2229] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:49:25.427366 7454 raft_consensus.cc:2258] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:49:25.428437 6614 catalog_manager.cc:4841] TS bfaefa9bf4394f328ea027a2c8787975 (127.27.54.67:33149): tablet 4a0ea3c009724a59aeed87a9149a3168 (table test-table [id=7a4b85420cd74f94bed7aac5b3c07e44]) successfully deleted
I20260807 06:49:25.429026 7454 ts_tablet_manager.cc:1905] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:49:25.433835 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 6745
I20260807 06:49:25.435429 7455 ts_tablet_manager.cc:1918] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:49:25.435706 7455 log.cc:1196] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/4a0ea3c009724a59aeed87a9149a3168
I20260807 06:49:25.436228 7455 ts_tablet_manager.cc:1939] T 4a0ea3c009724a59aeed87a9149a3168 P 8dc7b396e5814956a0b6034c6b45e158: Deleting consensus metadata
I20260807 06:49:25.437521 6611 catalog_manager.cc:4841] TS 8dc7b396e5814956a0b6034c6b45e158 (127.27.54.66:33063): tablet 4a0ea3c009724a59aeed87a9149a3168 (table test-table [id=7a4b85420cd74f94bed7aac5b3c07e44]) successfully deleted
I20260807 06:49:25.438067 7454 ts_tablet_manager.cc:1918] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 06:49:25.438282 7454 log.cc:1196] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/4a0ea3c009724a59aeed87a9149a3168
I20260807 06:49:25.438809 7454 ts_tablet_manager.cc:1939] T 4a0ea3c009724a59aeed87a9149a3168 P d0fb4917e8d64999bcc270395ba1f9b2: Deleting consensus metadata
I20260807 06:49:25.439924 6613 catalog_manager.cc:4841] TS d0fb4917e8d64999bcc270395ba1f9b2 (127.27.54.65:44713): tablet 4a0ea3c009724a59aeed87a9149a3168 (table test-table [id=7a4b85420cd74f94bed7aac5b3c07e44]) successfully deleted
Aug 07 06:49:25 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.65@KRBTEST.COM
I20260807 06:49:25.456748 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.435112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.27.54.65:44713)
0807 06:49:25.435331 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.435340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.438089 (+ 2749us) server_negotiation.cc:407] Connection header received
0807 06:49:25.438416 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.438427 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.438535 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.438683 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:25.440481 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.441407 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.443333 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.443767 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.443868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.448981 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:25.449003 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:25.449025 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:25.449076 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:25.453420 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:25.454197 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:25.454208 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:25.454212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:25.454417 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:25.454882 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:25.454888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:25.454891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:25.455432 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:25.455778 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.455995 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.456246 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:49:25.457516 6938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:25.627975 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 6745
I20260807 06:49:25.665808 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 6953
Aug 07 06:49:25 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.66@KRBTEST.COM
I20260807 06:49:25.682960 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.666722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.27.54.66:33063)
0807 06:49:25.667231 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.667236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.667253 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:49:25.667310 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.667314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.667375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.667467 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:25.668768 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.669529 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.670776 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.671389 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.671509 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.675856 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:25.675877 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:25.675897 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:25.675938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:25.679640 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:25.680409 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:25.680418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:25.680422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:25.680594 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:25.681029 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:25.681033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:25.681036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:25.681563 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:25.681908 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.682248 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.682478 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":290,"threads_started":1}
I20260807 06:49:25.683746 7144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:25.782286 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 6953
I20260807 06:49:25.810140 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 7162
Aug 07 06:49:25 dist-test-slave-3kk6 krb5kdc[6570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.54.67@KRBTEST.COM
I20260807 06:49:25.826849 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:25.811065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.27.54.67:33149)
0807 06:49:25.811250 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:25.811258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:25.811371 (+ 113us) server_negotiation.cc:407] Connection header received
0807 06:49:25.811617 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:25.811621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:25.811711 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:25.811825 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:25.813199 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.813905 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.815361 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:25.815835 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:25.815983 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:25.820764 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:25.820783 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:25.820799 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:25.820844 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:25.823822 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:25.824594 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:25.824602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:25.824605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:25.824743 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:25.825148 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:25.825154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:25.825159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:25.825532 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:25.825875 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:25.826172 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 06:49:25.826404 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:49:25.827528 7374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:25.956193 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 7162
I20260807 06:49:25.986835 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 6588
I20260807 06:49:25.988121 6693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:26.112835 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 6588
2026-08-07T06:49:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7682 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:49:26 dist-test-slave-3kk6 krb5kdc[7573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:49:26 dist-test-slave-3kk6 krb5kdc[7573](info): set up 2 sockets
Aug 07 06:49:26 dist-test-slave-3kk6 krb5kdc[7573](info): commencing operation
krb5kdc: starting...
W20260807 06:49:27.444410 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:49:27 dist-test-slave-3kk6 krb5kdc[7573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:49:27.755117 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41557
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:28.110059 7590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:28.110383 7590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:28.110569 7590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:28.117985 7590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:28.118111 7590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:28.118203 7590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:28.118316 7590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:28.118402 7590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:28.128835 7590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:28.130301 7590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:28.131703 7590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:28.144585 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:28.144515 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:28.144543 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:28.145490 7590 server_base.cc:1034] running on GCE node
I20260807 06:49:28.146363 7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:28.147948 7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:28.149188 7590 hybrid_clock.cc:648] HybridClock initialized: now 1786085368149143 us; error 60 us; skew 500 ppm
I20260807 06:49:28.149545 7590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T06:49:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2095 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:49:28 dist-test-slave-3kk6 krb5kdc[7610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:49:28 dist-test-slave-3kk6 krb5kdc[7610](info): set up 2 sockets
Aug 07 06:49:28 dist-test-slave-3kk6 krb5kdc[7610](info): commencing operation
krb5kdc: starting...
W20260807 06:49:29.537810 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.005s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:49:29 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:49:29.863232 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40025
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:30.231572 7624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:30.231947 7624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:30.232043 7624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:30.239686 7624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:30.239823 7624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:30.239880 7624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:30.239939 7624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:30.239987 7624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:30.250643 7624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:30.252154 7624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:30.253511 7624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:30.266275 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:30.266368 7629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:30.266319 7632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:30.267175 7624 server_base.cc:1034] running on GCE node
I20260807 06:49:30.268052 7624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:30.269570 7624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:30.270768 7624 hybrid_clock.cc:648] HybridClock initialized: now 1786085370270717 us; error 33 us; skew 500 ppm
I20260807 06:49:30.271163 7624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:30 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:30.274542 7624 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:49:30.276208 7624 webserver.cc:460] Webserver started at http://127.27.54.126:46593/ using document root <none> and password file <none>
I20260807 06:49:30.276908 7624 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:30.277014 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:30.277282 7624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:30.280445 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "bea95966dccb49069e087d8fb9f6b070"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "5a4ba949aeb71721d65950e1b2f18d93"
server_key_iv: "7b309d2ce97937062ab062d4254f5d7b"
server_key_version: "encryptionkey@0"
I20260807 06:49:30.281213 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "bea95966dccb49069e087d8fb9f6b070"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "5a4ba949aeb71721d65950e1b2f18d93"
server_key_iv: "7b309d2ce97937062ab062d4254f5d7b"
server_key_version: "encryptionkey@0"
I20260807 06:49:30.286746 7624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:49:30.290194 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:30.291654 7624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 06:49:30.291846 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "bea95966dccb49069e087d8fb9f6b070"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "5a4ba949aeb71721d65950e1b2f18d93"
server_key_iv: "7b309d2ce97937062ab062d4254f5d7b"
server_key_version: "encryptionkey@0"
I20260807 06:49:30.292112 7624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:30.308511 7624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:30.312779 7624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:30.313103 7624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:30.333284 7624 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40025
I20260807 06:49:30.333307 7713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40025 every 8 connection(s)
I20260807 06:49:30.334862 7624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:30.337689 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 7624
I20260807 06:49:30.337949 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:30.338616 27865 external_mini_cluster.cc:1428] Setting key 70618363849d3d0bfc737acb98dba7b9
I20260807 06:49:30.338922 7714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:30.354770 7714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Bootstrap starting.
Aug 07 06:49:30 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:30.358395 7714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:30.359575 7714 log.cc:824] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:30.362476 7714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: No bootstrap required, opened a new log
I20260807 06:49:30.363787 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:30.340964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.27.54.126:40025)
0807 06:49:30.341839 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:30.341850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:30.341900 (+ 50us) server_negotiation.cc:407] Connection header received
0807 06:49:30.344127 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:30.344149 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:30.344640 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:30.344976 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:30.347011 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:30.348350 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:30.350219 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:30.351011 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:30.351158 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:30.356222 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:30.356230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:30.356244 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:30.356291 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:30.359525 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:30.360308 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:30.360315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:30.360321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:30.360507 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:30.360911 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:30.360917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:30.360921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:30.361455 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:30.361751 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:30.362223 (+ 472us) server_negotiation.cc:299] Negotiation successful
0807 06:49:30.362815 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":447,"threads_started":1}
I20260807 06:49:30.371183 7714 raft_consensus.cc:348] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:30.371608 7714 raft_consensus.cc:374] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:30.371743 7714 raft_consensus.cc:729] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea95966dccb49069e087d8fb9f6b070, State: Initialized, Role: FOLLOWER
I20260807 06:49:30.372313 7714 consensus_queue.cc:260] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:30.372570 7714 raft_consensus.cc:388] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:30.372720 7714 raft_consensus.cc:482] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:30.372900 7714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:30.375316 7714 raft_consensus.cc:504] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:30.375874 7714 leader_election.cc:302] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bea95966dccb49069e087d8fb9f6b070; no voters:
I20260807 06:49:30.376524 7714 leader_election.cc:288] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:30.376634 7722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:30.377946 7722 raft_consensus.cc:686] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 LEADER]: Becoming Leader. State: Replica: bea95966dccb49069e087d8fb9f6b070, State: Running, Role: LEADER
I20260807 06:49:30.378717 7722 consensus_queue.cc:237] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:30.379354 7714 sys_catalog.cc:564] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:30.382046 7724 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bea95966dccb49069e087d8fb9f6b070. Latest consensus state: current_term: 1 leader_uuid: "bea95966dccb49069e087d8fb9f6b070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } } }
I20260807 06:49:30.382167 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bea95966dccb49069e087d8fb9f6b070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } } }
I20260807 06:49:30.382682 7724 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:30.382682 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:30.383617 7737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:30.389153 7737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:30.400468 7737 catalog_manager.cc:1347] Generated new cluster ID: 1ae8700451d945bb85ce0327c6ce7f89
I20260807 06:49:30.400678 7737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:30.434398 7737 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:30.435868 7737 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:30.442170 7737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Generated new TSK 0
I20260807 06:49:30.443135 7737 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:49:30.518926 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40025
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:30.876202 7754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:30.876523 7754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:30.876690 7754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:30.884642 7754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:30.884775 7754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:30.884960 7754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:30.895550 7754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:30.897082 7754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:30.898443 7754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:30.912153 7766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:30.912533 7764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:30.912688 7768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:30.913565 7754 server_base.cc:1034] running on GCE node
I20260807 06:49:30.914266 7754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:30.915449 7754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:30.916711 7754 hybrid_clock.cc:648] HybridClock initialized: now 1786085370916662 us; error 65 us; skew 500 ppm
I20260807 06:49:30.917061 7754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:30 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:30.920547 7754 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:49:30.922112 7754 webserver.cc:460] Webserver started at http://127.27.54.65:41389/ using document root <none> and password file <none>
I20260807 06:49:30.922870 7754 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:30.922987 7754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:30.923274 7754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:30.926290 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "924a1fb279c54368adc8c035ab0d8dad"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "3faeecb22abe8a4ec96282aa23dbc095"
server_key_iv: "66b2c60b2af3a3ebf44d6126b8b232ff"
server_key_version: "encryptionkey@0"
I20260807 06:49:30.927201 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "924a1fb279c54368adc8c035ab0d8dad"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "3faeecb22abe8a4ec96282aa23dbc095"
server_key_iv: "66b2c60b2af3a3ebf44d6126b8b232ff"
server_key_version: "encryptionkey@0"
I20260807 06:49:30.932773 7754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:49:30.936517 7778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:30.938091 7754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:49:30.938347 7754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "924a1fb279c54368adc8c035ab0d8dad"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "3faeecb22abe8a4ec96282aa23dbc095"
server_key_iv: "66b2c60b2af3a3ebf44d6126b8b232ff"
server_key_version: "encryptionkey@0"
I20260807 06:49:30.938609 7754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:30.965855 7754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:30.970449 7754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:30.970827 7754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:30.971931 7754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:30.973362 7754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:30.973443 7754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:30.973613 7754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:30.973675 7754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:31.008713 7754 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:45851
I20260807 06:49:31.008759 7946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:45851 every 8 connection(s)
I20260807 06:49:31.010149 7754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:31.012442 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 7754
I20260807 06:49:31.012706 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:31.013558 27865 external_mini_cluster.cc:1428] Setting key 1584c6980094a064e348a88009f1eabf
Aug 07 06:49:31 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085370, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:31.032907 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:31.013183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:56883 (local address 127.27.54.126:40025)
0807 06:49:31.013685 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:31.013694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:31.015148 (+ 1454us) server_negotiation.cc:407] Connection header received
0807 06:49:31.017261 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:31.017266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:31.017353 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:31.017480 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:31.019137 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.019862 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.021501 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.021924 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.022068 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:31.025887 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:31.025892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:31.025896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:31.025930 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:31.029229 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:31.030078 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:31.030087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:31.030093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:31.030260 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:31.030720 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:31.030728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:31.030735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:31.031136 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:31.031299 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:31.032098 (+ 799us) server_negotiation.cc:299] Negotiation successful
0807 06:49:31.032415 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
I20260807 06:49:31.033540 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:31.013939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:56883)
0807 06:49:31.014956 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:31.015014 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:31.016981 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:31.017732 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:31.017741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:31.018280 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:31.018949 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:31.018975 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.020049 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.020059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:31.021319 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:31.021333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.022173 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.022180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:31.022438 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:31.023270 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:31.023294 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:31.025613 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:31.029418 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:31.029426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:31.029440 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:31.029921 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:31.030378 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:31.030381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:31.030383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:31.030574 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:31.031269 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:31.031275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:31.031647 (+ 372us) client_negotiation.cc:769] Sending connection context
0807 06:49:31.031907 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 06:49:31.032325 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":381,"threads_started":1}
I20260807 06:49:31.035259 7947 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40025
I20260807 06:49:31.035661 7947 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:31.036644 7947 heartbeater.cc:511] Master 127.27.54.126:40025 requested a full tablet report, sending...
I20260807 06:49:31.039170 7670 ts_manager.cc:194] Registered new tserver with Master: 924a1fb279c54368adc8c035ab0d8dad (127.27.54.65:45851)
I20260807 06:49:31.040673 7670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:56883
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:49:31.085593 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40025
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:31.442739 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:31.443156 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:31.443331 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:31.451050 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:31.451221 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:31.451414 7954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:31.461823 7954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:31.463500 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:31.464910 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:31.480376 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:31.480346 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:31.480971 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:31.484445 7954 server_base.cc:1034] running on GCE node
I20260807 06:49:31.485178 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:31.486181 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:31.487398 7954 hybrid_clock.cc:648] HybridClock initialized: now 1786085371487357 us; error 38 us; skew 500 ppm
I20260807 06:49:31.487809 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:31 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:31.491400 7954 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:49:31.493003 7954 webserver.cc:460] Webserver started at http://127.27.54.66:41393/ using document root <none> and password file <none>
I20260807 06:49:31.493813 7954 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:31.493968 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:31.494325 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:31.497992 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "9bedd05fb03846e681730b5fa6ebe3d5"
format_stamp: "Formatted at 2026-08-07 06:49:31 on dist-test-slave-3kk6"
server_key: "d149335b3f1bb58bcf3d7002435bfd8a"
server_key_iv: "bdf2b94804b9007cf513430d5e6664f6"
server_key_version: "encryptionkey@0"
I20260807 06:49:31.499043 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "9bedd05fb03846e681730b5fa6ebe3d5"
format_stamp: "Formatted at 2026-08-07 06:49:31 on dist-test-slave-3kk6"
server_key: "d149335b3f1bb58bcf3d7002435bfd8a"
server_key_iv: "bdf2b94804b9007cf513430d5e6664f6"
server_key_version: "encryptionkey@0"
I20260807 06:49:31.504897 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:49:31.508767 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:31.510224 7954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:49:31.510437 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "9bedd05fb03846e681730b5fa6ebe3d5"
format_stamp: "Formatted at 2026-08-07 06:49:31 on dist-test-slave-3kk6"
server_key: "d149335b3f1bb58bcf3d7002435bfd8a"
server_key_iv: "bdf2b94804b9007cf513430d5e6664f6"
server_key_version: "encryptionkey@0"
I20260807 06:49:31.510696 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:31.541447 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:31.546641 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:31.547034 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:31.548141 7954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:31.549643 7954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:31.549755 7954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:31.549911 7954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:31.549996 7954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:31.587291 7954 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:37405
I20260807 06:49:31.587344 8132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:37405 every 8 connection(s)
I20260807 06:49:31.588862 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:31.589463 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 7954
I20260807 06:49:31.589676 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:31.590319 27865 external_mini_cluster.cc:1428] Setting key fb63197115319fa1e5175a286971d7a0
Aug 07 06:49:31 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:31.611980 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:31.591857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:60973 (local address 127.27.54.126:40025)
0807 06:49:31.592361 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:31.592369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:31.593596 (+ 1227us) server_negotiation.cc:407] Connection header received
0807 06:49:31.595753 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:31.595762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:31.595857 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:31.595986 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:31.597722 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.598970 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.600618 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.601392 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.601544 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:31.605238 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:31.605247 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:31.605254 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:31.605309 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:31.608261 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:31.609276 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:31.609280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:31.609282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:31.609414 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:31.609984 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:31.609988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:31.609990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:31.610309 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:31.610447 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:31.611498 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0807 06:49:31.611698 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":269,"threads_started":1}
I20260807 06:49:31.613693 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:31.592595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:60973)
0807 06:49:31.593426 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:31.593475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:31.595409 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:31.596205 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:31.596214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:31.596946 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:31.597533 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:31.597553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.599138 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.599142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:31.600452 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:31.600465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:31.601618 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:31.601622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:31.601829 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:31.602613 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:31.602637 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:31.604961 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:31.608427 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:31.608439 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:31.608459 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:31.609147 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:31.609542 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:31.609550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:31.609556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:31.609831 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:31.610474 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:31.610485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:31.611023 (+ 538us) client_negotiation.cc:769] Sending connection context
0807 06:49:31.611433 (+ 410us) client_negotiation.cc:240] Negotiation successful
0807 06:49:31.612070 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":380,"threads_started":1}
I20260807 06:49:31.615576 8133 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40025
I20260807 06:49:31.615928 8133 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:31.616706 8133 heartbeater.cc:511] Master 127.27.54.126:40025 requested a full tablet report, sending...
I20260807 06:49:31.618268 7670 ts_manager.cc:194] Registered new tserver with Master: 9bedd05fb03846e681730b5fa6ebe3d5 (127.27.54.66:37405)
I20260807 06:49:31.619288 7670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:60973
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:49:31.662537 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40025
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:32.018556 8141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:32.018970 8141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:32.019145 8141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:32.026707 8141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:32.026895 8141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:32.027107 8141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:32.037873 8141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:32.039422 8141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:32.040844 8141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 06:49:32.044824 7947 heartbeater.cc:503] Master 127.27.54.126:40025 was elected leader, sending a full tablet report...
W20260807 06:49:32.055830 8147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:32.055784 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:32.055867 8148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:32.056793 8141 server_base.cc:1034] running on GCE node
I20260807 06:49:32.057408 8141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:32.058357 8141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:32.059571 8141 hybrid_clock.cc:648] HybridClock initialized: now 1786085372059498 us; error 70 us; skew 500 ppm
I20260807 06:49:32.059926 8141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:32 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:32.063445 8141 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:49:32.065217 8141 webserver.cc:460] Webserver started at http://127.27.54.67:37121/ using document root <none> and password file <none>
I20260807 06:49:32.066021 8141 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:32.066183 8141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:32.066584 8141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:32.069751 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "b6a0c095b4d54a338482cff83858959b"
format_stamp: "Formatted at 2026-08-07 06:49:32 on dist-test-slave-3kk6"
server_key: "b9bcf7670b2425308029d3d9dac46291"
server_key_iv: "0d4404f6850819d014cf6e928670d803"
server_key_version: "encryptionkey@0"
I20260807 06:49:32.070641 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "b6a0c095b4d54a338482cff83858959b"
format_stamp: "Formatted at 2026-08-07 06:49:32 on dist-test-slave-3kk6"
server_key: "b9bcf7670b2425308029d3d9dac46291"
server_key_iv: "0d4404f6850819d014cf6e928670d803"
server_key_version: "encryptionkey@0"
I20260807 06:49:32.076686 8141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 06:49:32.081346 8162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:32.082906 8141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:32.083101 8141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "b6a0c095b4d54a338482cff83858959b"
format_stamp: "Formatted at 2026-08-07 06:49:32 on dist-test-slave-3kk6"
server_key: "b9bcf7670b2425308029d3d9dac46291"
server_key_iv: "0d4404f6850819d014cf6e928670d803"
server_key_version: "encryptionkey@0"
I20260807 06:49:32.083324 8141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:32.111750 8141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:32.116792 8141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:32.117129 8141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:32.118192 8141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:32.119792 8141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:32.119907 8141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:32.120062 8141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:32.120144 8141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:32.156047 8141 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:39355
I20260807 06:49:32.156111 8336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:39355 every 8 connection(s)
I20260807 06:49:32.157536 8141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:32.167719 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 8141
I20260807 06:49:32.167943 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:49:32.168408 27865 external_mini_cluster.cc:1428] Setting key 9396dd4d210e0f1aaa03f9f3f0ee48bb
Aug 07 06:49:32 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:32.179047 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:32.160349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:46305 (local address 127.27.54.126:40025)
0807 06:49:32.160884 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:32.160890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:32.161911 (+ 1021us) server_negotiation.cc:407] Connection header received
0807 06:49:32.164008 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:32.164014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:32.164112 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:32.164244 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:32.165913 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.166825 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.168627 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.169037 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.169186 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:32.172887 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:32.172893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:32.172896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:32.172929 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:32.175750 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.176553 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.176559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.176563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.176705 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.177162 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.177170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.177175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.177457 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:32.177630 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:32.178341 (+ 711us) server_negotiation.cc:299] Negotiation successful
0807 06:49:32.178585 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":247,"threads_started":1}
I20260807 06:49:32.180121 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:32.160958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:46305)
0807 06:49:32.161713 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:32.161761 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:32.163700 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:32.164517 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:32.164526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:32.165099 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:32.165715 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:32.165735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.166989 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.166999 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:32.168426 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:32.168440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.169293 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.169303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:32.169552 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:32.170298 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:32.170322 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:32.172658 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:32.175920 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:32.175930 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:32.175949 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:32.176420 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:32.176826 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:32.176831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:32.176834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:32.177043 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:32.177604 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:32.177613 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:32.177966 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 06:49:32.178259 (+ 293us) client_negotiation.cc:240] Negotiation successful
0807 06:49:32.178860 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":293,"threads_started":1}
I20260807 06:49:32.181702 8338 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40025
I20260807 06:49:32.182037 8338 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:32.183003 8338 heartbeater.cc:511] Master 127.27.54.126:40025 requested a full tablet report, sending...
I20260807 06:49:32.184571 7670 ts_manager.cc:194] Registered new tserver with Master: b6a0c095b4d54a338482cff83858959b (127.27.54.67:39355)
I20260807 06:49:32.185400 7670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:46305
I20260807 06:49:32.196808 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab.
Entry for principal oryx/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab.
WARNING: no policy specified for oryx/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab.
Entry for principal oryx/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab.
WARNING: no policy specified for oryx/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab.
Entry for principal oryx/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab.
I20260807 06:49:32.394023 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 7754
Aug 07 06:49:32 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:49:32.411914 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:32.395443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.27.54.65:45851)
0807 06:49:32.396206 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:32.396213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:32.396254 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:49:32.396332 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:32.396339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:32.396556 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:32.396768 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:32.398308 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.399415 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.400740 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.401355 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.401469 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:32.405886 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:32.405900 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:32.405914 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:32.405953 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:32.409049 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.409792 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.409799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.409801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.409937 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.410309 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.410312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.410314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.410736 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:32.410980 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:32.411247 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 06:49:32.411469 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":423,"threads_started":1}
I20260807 06:49:32.413532 7940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:32.501910 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 7754
I20260807 06:49:32.529091 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 7954
Aug 07 06:49:32 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:49:32.546900 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:32.530114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.27.54.66:37405)
0807 06:49:32.530717 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:32.530728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:32.530763 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:49:32.530844 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:32.530851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:32.531059 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:32.531237 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:32.532646 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.533700 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.535077 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.535643 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.535793 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:32.540479 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:32.540494 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:32.540509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:32.540551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:32.543748 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.544523 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.544531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.544533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.544665 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.545080 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.545085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.545088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.545468 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:32.545792 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:32.546143 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 06:49:32.546429 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260807 06:49:32.548252 8127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:32.632840 8133 heartbeater.cc:503] Master 127.27.54.126:40025 was elected leader, sending a full tablet report...
I20260807 06:49:32.634853 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 7954
I20260807 06:49:32.657382 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8141
Aug 07 06:49:32 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:49:32.675159 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:32.658498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.27.54.67:39355)
0807 06:49:32.659129 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:32.659137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:32.659168 (+ 31us) server_negotiation.cc:407] Connection header received
0807 06:49:32.659292 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:32.659304 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:32.659535 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:32.659667 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:32.661091 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.662122 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.663616 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:32.664195 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:32.664304 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:32.669128 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:32.669145 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:32.669162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:32.669206 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:32.672273 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.673058 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.673065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.673068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.673194 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:32.673550 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:32.673554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:32.673556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:32.673930 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:32.674160 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:32.674432 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 06:49:32.674646 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":335,"threads_started":1}
I20260807 06:49:32.676546 8330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:32.762950 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 8141
I20260807 06:49:32.786574 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 7624
I20260807 06:49:32.788028 7709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:32.895422 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 7624
I20260807 06:49:32.923143 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40025
--webserver_interface=127.27.54.126
--webserver_port=46593
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:33.278435 8359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:33.278901 8359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:33.279072 8359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:33.286839 8359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:33.286989 8359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:33.287091 8359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:33.287195 8359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:33.287278 8359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:33.297537 8359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=46593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:33.299062 8359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:33.300441 8359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:33.312817 8370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:33.312801 8368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:33.312882 8372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:33.313666 8359 server_base.cc:1034] running on GCE node
I20260807 06:49:33.314496 8359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:33.316001 8359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:33.317241 8359 hybrid_clock.cc:648] HybridClock initialized: now 1786085373317203 us; error 51 us; skew 500 ppm
I20260807 06:49:33.317606 8359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:33 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085373, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:33.324383 8359 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:49:33.326025 8359 webserver.cc:460] Webserver started at http://127.27.54.126:46593/ using document root <none> and password file <none>
I20260807 06:49:33.326833 8359 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:33.327073 8359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:33.332782 8359 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:49:33.335788 8382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:33.337198 8359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:49:33.337399 8359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "bea95966dccb49069e087d8fb9f6b070"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "5a4ba949aeb71721d65950e1b2f18d93"
server_key_iv: "7b309d2ce97937062ab062d4254f5d7b"
server_key_version: "encryptionkey@0"
I20260807 06:49:33.338452 8359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:33.379313 8359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:33.383733 8359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:33.383998 8359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:33.404219 8359 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40025
I20260807 06:49:33.404238 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40025 every 8 connection(s)
I20260807 06:49:33.406004 8359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:33.413592 8472 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:49:33.415901 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 8359
I20260807 06:49:33.416751 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:45851
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:40025
--webserver_port=41389
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:33.417382 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Bootstrap starting.
I20260807 06:49:33.437412 8472 log.cc:824] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:33.447219 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:49:33.447829 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Bootstrap complete.
I20260807 06:49:33.457506 8472 raft_consensus.cc:348] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:33.458438 8472 raft_consensus.cc:729] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea95966dccb49069e087d8fb9f6b070, State: Initialized, Role: FOLLOWER
I20260807 06:49:33.459151 8472 consensus_queue.cc:260] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:33.459437 8472 raft_consensus.cc:388] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:33.459570 8472 raft_consensus.cc:482] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:33.459776 8472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:49:33.464008 8472 raft_consensus.cc:504] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:33.464570 8472 leader_election.cc:302] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bea95966dccb49069e087d8fb9f6b070; no voters:
I20260807 06:49:33.465250 8472 leader_election.cc:288] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:49:33.465443 8482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:49:33.466924 8482 raft_consensus.cc:686] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [term 2 LEADER]: Becoming Leader. State: Replica: bea95966dccb49069e087d8fb9f6b070, State: Running, Role: LEADER
I20260807 06:49:33.467666 8482 consensus_queue.cc:237] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } }
I20260807 06:49:33.468760 8472 sys_catalog.cc:564] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:33.470534 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bea95966dccb49069e087d8fb9f6b070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } } }
I20260807 06:49:33.470973 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:33.471563 8483 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bea95966dccb49069e087d8fb9f6b070. Latest consensus state: current_term: 2 leader_uuid: "bea95966dccb49069e087d8fb9f6b070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea95966dccb49069e087d8fb9f6b070" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40025 } } }
I20260807 06:49:33.471905 8483 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:33.479950 8489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:33.485878 8489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:33.487951 8489 catalog_manager.cc:1259] Loaded cluster ID: 1ae8700451d945bb85ce0327c6ce7f89
I20260807 06:49:33.488068 8489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:33.492333 8489 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:33.493877 8489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bea95966dccb49069e087d8fb9f6b070: Loaded TSK: 0
I20260807 06:49:33.495493 8489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:49:33.787492 8474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:33.787876 8474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:33.788043 8474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:33.795691 8474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:33.795884 8474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:33.796072 8474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:33.806304 8474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.65.keytab
--principal=oryx/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:45851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=41389
--tserver_master_addrs=127.27.54.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:33.807932 8474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:33.809294 8474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:33.824299 8510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:33.824352 8512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:33.825874 8515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:33.826298 8474 server_base.cc:1034] running on GCE node
I20260807 06:49:33.827061 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:33.828173 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:33.829406 8474 hybrid_clock.cc:648] HybridClock initialized: now 1786085373829361 us; error 40 us; skew 500 ppm
I20260807 06:49:33.829747 8474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:33 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085373, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:33.833176 8474 init.cc:377] Logged in from keytab as oryx/127.27.54.65@KRBTEST.COM (short username oryx)
I20260807 06:49:33.834975 8474 webserver.cc:460] Webserver started at http://127.27.54.65:41389/ using document root <none> and password file <none>
I20260807 06:49:33.835716 8474 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:33.835902 8474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:33.841662 8474 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:49:33.844641 8526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:33.846066 8474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:49:33.846258 8474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "924a1fb279c54368adc8c035ab0d8dad"
format_stamp: "Formatted at 2026-08-07 06:49:30 on dist-test-slave-3kk6"
server_key: "3faeecb22abe8a4ec96282aa23dbc095"
server_key_iv: "66b2c60b2af3a3ebf44d6126b8b232ff"
server_key_version: "encryptionkey@0"
I20260807 06:49:33.847389 8474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:33.877929 8474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:33.882301 8474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:33.882556 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:33.883636 8474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:33.884994 8474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:33.885092 8474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:33.885174 8474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:33.885229 8474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:33.920661 8474 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:45851
I20260807 06:49:33.920754 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:45851 every 8 connection(s)
I20260807 06:49:33.922190 8474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:33.929790 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 8474
I20260807 06:49:33.930612 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:37405
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:40025
--webserver_port=41393
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:49:33 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:33 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:49:34.307210 8707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:34.307576 8707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:34.307753 8707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:34.315616 8707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:34.315804 8707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:34.315990 8707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:34.326319 8707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.66.keytab
--principal=oryx/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:37405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=41393
--tserver_master_addrs=127.27.54.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:34.327831 8707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:34.329280 8707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:34.350446 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:34.350404 8718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:34.351787 8716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:34.352128 8707 server_base.cc:1034] running on GCE node
I20260807 06:49:34.353111 8707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:34.354339 8707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:34.357764 8707 hybrid_clock.cc:648] HybridClock initialized: now 1786085374357657 us; error 114 us; skew 500 ppm
I20260807 06:49:34.358227 8707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:34 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085374, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:34.362463 8707 init.cc:377] Logged in from keytab as oryx/127.27.54.66@KRBTEST.COM (short username oryx)
I20260807 06:49:34.364535 8707 webserver.cc:460] Webserver started at http://127.27.54.66:41393/ using document root <none> and password file <none>
I20260807 06:49:34.365366 8707 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:34.365545 8707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:34.373191 8707 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 06:49:34.377527 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:34.379540 8707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 06:49:34.379765 8707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "9bedd05fb03846e681730b5fa6ebe3d5"
format_stamp: "Formatted at 2026-08-07 06:49:31 on dist-test-slave-3kk6"
server_key: "d149335b3f1bb58bcf3d7002435bfd8a"
server_key_iv: "bdf2b94804b9007cf513430d5e6664f6"
server_key_version: "encryptionkey@0"
I20260807 06:49:34.380908 8707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:34.398625 8707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:34.403600 8707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:34.403998 8707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:34.405215 8707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:34.406906 8707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:34.407032 8707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:34.407184 8707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:34.407284 8707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:34.458007 8707 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:37405
I20260807 06:49:34.458065 8892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:37405 every 8 connection(s)
I20260807 06:49:34.459712 8707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:34.469473 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 8707
I20260807 06:49:34.470151 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:39355
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:40025
--webserver_port=37121
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:49:34 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:34 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:49:35.140527 8706 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.203s user 0.005s sys 0.013s
I20260807 06:49:35.142939 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:33.925324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:39701)
0807 06:49:33.926161 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:33.926199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:33.928010 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:33.929459 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:33.929470 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:33.930294 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:33.931168 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:33.931197 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:33.932874 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:33.932882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:33.935113 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:33.935146 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:33.936204 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:33.936214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:33.936544 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:33.937376 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:33.937413 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:35.141508 (+1204095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":382,"threads_started":1}
W20260807 06:49:35.143151 8705 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:33.925284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:39701 (local address 127.27.54.126:40025)
0807 06:49:33.926118 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:33.926134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:33.926337 (+ 203us) server_negotiation.cc:407] Connection header received
0807 06:49:33.928553 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:33.928566 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:33.928903 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:33.929205 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:33.931402 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:33.932683 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:33.935279 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:33.935949 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:33.936091 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.142258 (+1206167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:39701: BlockingRecv error: recv got EOF from 127.27.54.65:39701 (error 108)
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":300,"threads_started":1}
W20260807 06:49:35.144400 8699 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40025 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.54.126:40025: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:35.162962 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:35.145213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:43151)
0807 06:49:35.145485 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:35.145506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:35.145648 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:35.146270 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:35.146276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:35.146834 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:35.147105 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.147123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.151762 (+ 4639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.151769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.153750 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.153783 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.155622 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.155633 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.155924 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.157163 (+ 1239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:35.157195 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:35.162575 (+ 5380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159}
W20260807 06:49:35.163170 8705 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:35.145288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:43151 (local address 127.27.54.126:40025)
0807 06:49:35.145464 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:35.145471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:35.145794 (+ 323us) server_negotiation.cc:407] Connection header received
0807 06:49:35.145868 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:35.145873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:35.145952 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:35.146054 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:35.147316 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.151383 (+ 4067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.154023 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.155038 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.155195 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.162888 (+ 7693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:43151: BlockingRecv error: recv got EOF from 127.27.54.65:43151 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:35.173699 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:35.164180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:40239)
0807 06:49:35.164444 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:35.164464 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:35.164594 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:35.165012 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:35.165017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:35.165374 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:35.165599 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.165608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.166735 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.166741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.167767 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.167780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.168538 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.168544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.168775 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.169546 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:35.169565 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:35.173265 (+ 3700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260807 06:49:35.173736 8705 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:35.164315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:40239 (local address 127.27.54.126:40025)
0807 06:49:35.164487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:35.164494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:35.164518 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:49:35.164793 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:35.164800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:35.164870 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:35.164981 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:35.165759 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.166534 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.167923 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.168370 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.168521 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.173510 (+ 4989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:40239: BlockingRecv error: recv got EOF from 127.27.54.65:40239 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 06:49:35.174358 8699 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:49:35.246138 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:35.246820 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:35.247087 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:35.266779 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:35.267069 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:35.267382 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:35.293689 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/krb5kdc/oryx_127.27.54.67.keytab
--principal=oryx/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:39355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=37121
--tserver_master_addrs=127.27.54.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:35.296437 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:35.299199 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:35.320571 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:35.320571 8909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:35.321179 8908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:35.322567 8902 server_base.cc:1034] running on GCE node
I20260807 06:49:35.323817 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:35.325722 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:35.327154 8902 hybrid_clock.cc:648] HybridClock initialized: now 1786085375327052 us; error 76 us; skew 500 ppm
I20260807 06:49:35.327775 8902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085375, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:35.333869 8902 init.cc:377] Logged in from keytab as oryx/127.27.54.67@KRBTEST.COM (short username oryx)
I20260807 06:49:35.336618 8902 webserver.cc:460] Webserver started at http://127.27.54.67:37121/ using document root <none> and password file <none>
I20260807 06:49:35.338091 8902 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:35.338359 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:35.349737 8902 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260807 06:49:35.354974 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:35.357712 8902 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:49:35.358044 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "b6a0c095b4d54a338482cff83858959b"
format_stamp: "Formatted at 2026-08-07 06:49:32 on dist-test-slave-3kk6"
server_key: "b9bcf7670b2425308029d3d9dac46291"
server_key_iv: "0d4404f6850819d014cf6e928670d803"
server_key_version: "encryptionkey@0"
I20260807 06:49:35.360411 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:35.404507 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:35.413312 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:35.413800 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:35.415720 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:35.418517 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:35.418718 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:35.418897 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:35.419018 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:35.465571 8902 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:39355
I20260807 06:49:35.465620 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:39355 every 8 connection(s)
I20260807 06:49:35.468492 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:35.470335 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 8902
I20260807 06:49:35.499141 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:35.472119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.27.54.126:40025)
0807 06:49:35.472496 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:35.472502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:35.473218 (+ 716us) server_negotiation.cc:407] Connection header received
0807 06:49:35.473499 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:35.473508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:35.473619 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:35.473762 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:35.477858 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.479409 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.483933 (+ 4524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.484631 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.484774 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.490221 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:35.490231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:35.490250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:35.490299 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:35.494726 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:35.496014 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:35.496026 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:35.496036 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:35.496330 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:35.496921 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:35.496928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:35.496934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:35.497490 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:35.497838 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:35.498382 (+ 544us) server_negotiation.cc:299] Negotiation successful
0807 06:49:35.498847 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:49:35.707880 8900 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.231s user 0.005s sys 0.013s
W20260807 06:49:35.709724 8899 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:34.462950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:35375 (local address 127.27.54.126:40025)
0807 06:49:34.463725 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:34.463736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:34.464647 (+ 911us) server_negotiation.cc:407] Connection header received
0807 06:49:34.466643 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:34.466672 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:34.466786 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:34.466983 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:34.468577 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:34.470438 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:34.473396 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:34.474465 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:34.474738 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.709453 (+1234715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:35375: BlockingRecv error: recv got EOF from 127.27.54.66:35375 (error 108)
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":361,"threads_started":1}
I20260807 06:49:35.711120 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:34.463536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:35375)
0807 06:49:34.464425 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:34.464495 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:34.466328 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:34.467181 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:34.467191 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:34.467765 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:34.468342 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:34.468371 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:34.470694 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:34.470702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:34.473137 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:34.473161 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:34.474839 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:34.474858 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:34.475241 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:34.476422 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:34.476461 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:35.709291 (+1232830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":380,"threads_started":1}
W20260807 06:49:35.712354 8894 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40025 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.54.126:40025: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 06:49:35.724040 8899 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:35.713362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:57939 (local address 127.27.54.126:40025)
0807 06:49:35.713543 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:35.713549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:35.713572 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:49:35.713693 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:35.713699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:35.713772 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:35.713886 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:35.714714 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.715526 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.716916 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.717462 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.717615 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.723851 (+ 6236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:57939: BlockingRecv error: recv got EOF from 127.27.54.66:57939 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:49:35.724134 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:35.713230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:57939)
0807 06:49:35.713410 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:35.713435 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:35.713572 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:35.713925 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:35.713930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:35.714348 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:35.714526 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.714535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.715691 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.715697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.716727 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.716741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.717886 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.717892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.718077 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.718876 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:35.718894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:35.723662 (+ 4768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:35 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:35.738715 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:35.725091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:44099)
0807 06:49:35.725360 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:35.725394 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:35.725567 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:35.725997 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:35.726005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:35.726718 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:35.727048 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.727067 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.728209 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.728218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.730759 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.730786 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.732253 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.732264 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.732614 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.733489 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:35.733526 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:35.738350 (+ 4824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260807 06:49:35.738940 8899 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:35.725359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:44099 (local address 127.27.54.126:40025)
0807 06:49:35.725530 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:35.725534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:35.725552 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:49:35.725710 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:35.725714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:35.725807 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:35.725890 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:35.727234 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.728022 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.731257 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.731843 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.731973 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.738602 (+ 6629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:44099: BlockingRecv error: recv got EOF from 127.27.54.66:44099 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 06:49:35.739151 8894 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:36.186815 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:36.175111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:51433)
0807 06:49:36.175782 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:36.175811 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:36.176004 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:36.176425 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:36.176430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:36.176927 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:36.177126 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.177139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.178269 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.178275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.179379 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.179393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.180190 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.180196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.180385 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.181097 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:36.181116 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:36.186424 (+ 5308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":422,"threads_started":1}
W20260807 06:49:36.186951 8899 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:36.175287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51433 (local address 127.27.54.126:40025)
0807 06:49:36.175560 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:36.175571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:36.175913 (+ 342us) server_negotiation.cc:407] Connection header received
0807 06:49:36.176166 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:36.176171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:36.176256 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:36.176382 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:36.177313 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.178101 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.179543 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.179987 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.180132 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.186704 (+ 6572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:51433: BlockingRecv error: recv got EOF from 127.27.54.65:51433 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260807 06:49:36.623611 9113 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.126s user 0.005s sys 0.025s
W20260807 06:49:36.625020 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:35.472256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:57837 (local address 127.27.54.126:40025)
0807 06:49:35.487447 (+ 15191us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:35.487453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:35.487474 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:49:35.488475 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:35.488480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:35.488567 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:35.488707 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:35.490680 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.492728 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.495277 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.495703 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.495846 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.624711 (+1128865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:57837: BlockingRecv error: recv got EOF from 127.27.54.67:57837 (error 108)
Metrics: {"server-negotiator.queue_time_us":15067,"thread_start_us":15236,"threads_started":1}
I20260807 06:49:36.625631 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:35.472692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:57837)
0807 06:49:35.485332 (+ 12640us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:35.485416 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:35.488168 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:35.489021 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:35.489030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:35.489598 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:35.490361 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.490390 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.492966 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.492972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.495119 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:35.495132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:35.495961 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:35.495972 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:35.496300 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:35.497241 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:35.497292 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:36.624496 (+1127204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11031,"thread_start_us":11143,"threads_started":1}
W20260807 06:49:36.626636 9106 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40025 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.54.126:40025: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:36.636988 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:36.627437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:59745)
0807 06:49:36.627631 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:36.627648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:36.627754 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:36.628066 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:36.628069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:36.628444 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:36.628625 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.628634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.629730 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.629735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.630830 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.630845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.631534 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.631539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.631673 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.632392 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:36.632410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:36.636751 (+ 4341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260807 06:49:36.637189 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:36.627523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:59745 (local address 127.27.54.126:40025)
0807 06:49:36.627804 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:36.627808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:36.627826 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:49:36.627878 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:36.627881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:36.627941 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:36.628022 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:36.628796 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.629582 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.630967 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.631371 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.631472 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.636972 (+ 5500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:59745: BlockingRecv error: recv got EOF from 127.27.54.67:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:36.646454 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:36.637669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:40651)
0807 06:49:36.637850 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:36.637870 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:36.637973 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:36.638237 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:36.638240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:36.638551 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:36.638815 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.638826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.639794 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.639798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.640775 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.640788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.641521 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.641525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.641655 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.642328 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:36.642345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:36.646126 (+ 3781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 06:49:36.646863 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:36.637771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:40651 (local address 127.27.54.126:40025)
0807 06:49:36.637926 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:36.637931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:36.637946 (+ 15us) server_negotiation.cc:407] Connection header received
0807 06:49:36.638054 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:36.638057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:36.638113 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:36.638187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:36.638942 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.639648 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.640914 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.641341 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.641434 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.646370 (+ 4936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:40651: BlockingRecv error: recv got EOF from 127.27.54.67:40651 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 06:49:36.646864 9106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:36 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:36.750165 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:36.739719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:46661)
0807 06:49:36.740382 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:36.740404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:36.740530 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:36.740884 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:36.740887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:36.741365 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:36.741520 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.741529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.742885 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.742889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.743942 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:36.743955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.744683 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.744687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:36.744807 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.745560 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:36.745577 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:36.749906 (+ 4329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":411,"threads_started":1}
W20260807 06:49:36.750381 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:36.739900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:46661 (local address 127.27.54.126:40025)
0807 06:49:36.740110 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:36.740115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:36.740490 (+ 375us) server_negotiation.cc:407] Connection header received
0807 06:49:36.740639 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:36.740645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:36.740742 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:36.740847 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:36.741718 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.742735 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.744078 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:36.744513 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:36.744623 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:36.750131 (+ 5508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:46661: BlockingRecv error: recv got EOF from 127.27.54.66:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 06:49:37 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:37 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:37.197954 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:37.187921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:48287)
0807 06:49:37.188500 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:37.188517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:37.188662 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:37.188998 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:37.189002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:37.189433 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:37.189628 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:37.189636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.190825 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.190829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:37.191805 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:37.191818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.192553 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.192557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:37.192691 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:37.193356 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:37.193375 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:37.197606 (+ 4231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":344,"threads_started":1}
W20260807 06:49:37.198199 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:37.188042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:48287 (local address 127.27.54.126:40025)
0807 06:49:37.188222 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:37.188226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:37.188602 (+ 376us) server_negotiation.cc:407] Connection header received
0807 06:49:37.188778 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:37.188782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:37.188856 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:37.188945 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:37.189772 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.190595 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.191965 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.192404 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.192507 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:37.197908 (+ 5401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:48287: BlockingRecv error: recv got EOF from 127.27.54.65:48287 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 06:49:37 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:37 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:37.657263 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:37.647491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:52469)
0807 06:49:37.648112 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:37.648129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:37.648252 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:37.648582 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:37.648585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:37.649039 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:37.649190 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:37.649198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.650225 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.650230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:37.651339 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:37.651365 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.652147 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.652151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:37.652276 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:37.652942 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:37.652960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:37.656896 (+ 3936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":369,"threads_started":1}
W20260807 06:49:37.657430 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:37.647644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:52469 (local address 127.27.54.126:40025)
0807 06:49:37.647836 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:37.647840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:37.648208 (+ 368us) server_negotiation.cc:407] Connection header received
0807 06:49:37.648360 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:37.648363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:37.648442 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:37.648530 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:37.649311 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.650058 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.651499 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.651975 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.652075 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:37.657144 (+ 5069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:52469: BlockingRecv error: recv got EOF from 127.27.54.67:52469 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 06:49:37 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:37 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:37.761279 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:37.751264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:56351)
0807 06:49:37.751943 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:37.751969 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:37.752141 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:37.752498 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:37.752503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:37.752958 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:37.753147 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:37.753156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.754217 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.754223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:37.755203 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:37.755215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.756087 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.756093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:37.756288 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:37.757161 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:37.757179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:37.760867 (+ 3688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":325,"threads_started":1}
W20260807 06:49:37.761432 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:37.751581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:56351 (local address 127.27.54.126:40025)
0807 06:49:37.751846 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:37.751865 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:37.752067 (+ 202us) server_negotiation.cc:407] Connection header received
0807 06:49:37.752275 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:37.752285 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:37.752352 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:37.752459 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:37.753310 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.754062 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.755363 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:37.755914 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:37.756062 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:37.761140 (+ 5078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:56351: BlockingRecv error: recv got EOF from 127.27.54.66:56351 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 07 06:49:38 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:38 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:38.209117 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:38.199238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:36005)
0807 06:49:38.199898 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:38.199916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:38.200097 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:38.200478 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:38.200483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:38.200905 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:38.201091 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:38.201100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.202179 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.202186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:38.203238 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:38.203252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.204004 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.204010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:38.204206 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:38.204958 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:38.204976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:38.208735 (+ 3759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":338,"threads_started":1}
W20260807 06:49:38.209218 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:38.199560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:36005 (local address 127.27.54.126:40025)
0807 06:49:38.199825 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:38.199832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:38.200014 (+ 182us) server_negotiation.cc:407] Connection header received
0807 06:49:38.200244 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:38.200250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:38.200327 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:38.200440 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:38.201261 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.202032 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.203400 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.203832 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.203992 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:38.208984 (+ 4992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:36005: BlockingRecv error: recv got EOF from 127.27.54.65:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 07 06:49:38 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:38 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:38.672247 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:38.658931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:34635)
0807 06:49:38.659433 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:38.659450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:38.659590 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:38.659947 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:38.659950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:38.660364 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:38.660589 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:38.660599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.661605 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.661609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:38.662920 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:38.662933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.664087 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.664093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:38.664281 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:38.667911 (+ 3630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:38.667930 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:38.671871 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":298,"threads_started":1}
W20260807 06:49:38.672374 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:38.659003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:34635 (local address 127.27.54.126:40025)
0807 06:49:38.659264 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:38.659271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:38.659546 (+ 275us) server_negotiation.cc:407] Connection header received
0807 06:49:38.659716 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:38.659720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:38.659796 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:38.659900 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:38.660732 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.661482 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.663064 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.663921 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.664067 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:38.672080 (+ 8013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:34635: BlockingRecv error: recv got EOF from 127.27.54.67:34635 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 06:49:38 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:38 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:38.771577 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:38.762347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:59657)
0807 06:49:38.762897 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:38.762919 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:38.763037 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:38.763330 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:38.763333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:38.763784 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:38.763967 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:38.763977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.765101 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.765105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:38.766061 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:38.766073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.766817 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.766821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:38.766951 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:38.767570 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:38.767588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:38.771308 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":327,"threads_started":1}
W20260807 06:49:38.771852 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:38.762389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:59657 (local address 127.27.54.126:40025)
0807 06:49:38.762615 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:38.762625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:38.762998 (+ 373us) server_negotiation.cc:407] Connection header received
0807 06:49:38.763136 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:38.763139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:38.763205 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:38.763286 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:38.764100 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.764977 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.766188 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:38.766685 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:38.766781 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:38.771561 (+ 4780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:59657: BlockingRecv error: recv got EOF from 127.27.54.66:59657 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 06:49:39 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:39 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:39.220852 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:39.210389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:45199)
0807 06:49:39.210998 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:39.211014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:39.211184 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:39.211552 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:39.211557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:39.211999 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:39.212170 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:39.212178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.213722 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.213727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:39.214826 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:39.214840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.215576 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.215582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:39.215779 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:39.216531 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:39.216549 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:39.220467 (+ 3918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":373,"threads_started":1}
W20260807 06:49:39.221021 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:39.210476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:45199 (local address 127.27.54.126:40025)
0807 06:49:39.210756 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:39.210763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:39.211119 (+ 356us) server_negotiation.cc:407] Connection header received
0807 06:49:39.211331 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:39.211336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:39.211403 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:39.211512 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:39.212321 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.213576 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.214980 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.215401 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.215547 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:39.220726 (+ 5179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:45199: BlockingRecv error: recv got EOF from 127.27.54.65:45199 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 07 06:49:39 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:39 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:39.682771 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:39.673226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:41861)
0807 06:49:39.673719 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:39.673737 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:39.673859 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:39.674182 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:39.674185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:39.674566 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:39.674783 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:39.674792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.675788 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.675791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:39.676804 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:39.676817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.677571 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.677575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:39.677699 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:39.678465 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:39.678482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:39.682376 (+ 3894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
W20260807 06:49:39.682938 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:39.673294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:41861 (local address 127.27.54.126:40025)
0807 06:49:39.673555 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:39.673563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:39.673826 (+ 263us) server_negotiation.cc:407] Connection header received
0807 06:49:39.673963 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:39.673967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:39.674047 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:39.674138 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:39.674917 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.675668 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.676934 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.677438 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.677536 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:39.682630 (+ 5094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:41861: BlockingRecv error: recv got EOF from 127.27.54.67:41861 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 07 06:49:39 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:39 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:39.782555 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:39.772689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:53657)
0807 06:49:39.773251 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:39.773271 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:39.773453 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:39.773809 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:39.773813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:39.774259 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:39.774445 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:39.774454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.775617 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.775628 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:39.776775 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:39.776788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.777548 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.777555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:39.777742 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:39.778463 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:39.778481 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:39.782226 (+ 3745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":308,"threads_started":1}
W20260807 06:49:39.782938 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:39.772789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:53657 (local address 127.27.54.126:40025)
0807 06:49:39.773060 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:39.773067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:39.773376 (+ 309us) server_negotiation.cc:407] Connection header received
0807 06:49:39.773587 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:39.773593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:39.773668 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:39.773774 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:39.774602 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.775464 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.776933 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:39.777389 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:39.777531 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:39.782575 (+ 5044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:53657: BlockingRecv error: recv got EOF from 127.27.54.66:53657 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 07 06:49:40 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:40 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:40.231688 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:40.222224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:57677)
0807 06:49:40.222890 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:40.222913 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:40.223100 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:40.223498 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:40.223502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:40.224140 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:40.224327 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:40.224336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.225353 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.225358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:40.226323 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:40.226336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.227036 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.227041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:40.227174 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:40.227806 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:40.227823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:40.231443 (+ 3620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":546,"threads_started":1}
W20260807 06:49:40.231945 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:40.222398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:57677 (local address 127.27.54.126:40025)
0807 06:49:40.222613 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:40.222617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:40.223006 (+ 389us) server_negotiation.cc:407] Connection header received
0807 06:49:40.223245 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:40.223249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:40.223315 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:40.223452 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:40.224470 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.225218 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.226461 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.226903 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.227000 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:40.231710 (+ 4710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:57677: BlockingRecv error: recv got EOF from 127.27.54.65:57677 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 06:49:40 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:40 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:40.693892 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:40.684242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:53851)
0807 06:49:40.684944 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:40.684970 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:40.685123 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:40.685567 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:40.685571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:40.686154 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:40.686420 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:40.686433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.687647 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.687650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:40.688656 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:40.688666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.689317 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.689321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:40.689452 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:40.690096 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:40.690112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:40.693616 (+ 3504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":371,"threads_started":1}
W20260807 06:49:40.694124 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:40.684379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:53851 (local address 127.27.54.126:40025)
0807 06:49:40.684692 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:40.684699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:40.685089 (+ 390us) server_negotiation.cc:407] Connection header received
0807 06:49:40.685276 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:40.685280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:40.685375 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:40.685501 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:40.686600 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.687512 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.688781 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.689182 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.689275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:40.693852 (+ 4577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:53851: BlockingRecv error: recv got EOF from 127.27.54.67:53851 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 07 06:49:40 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:40 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:40.793756 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:40.784035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:43193)
0807 06:49:40.784851 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:40.784874 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:40.784995 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:40.785359 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:40.785364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:40.785953 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:40.786136 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:40.786147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.787281 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.787287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:40.788275 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:40.788288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.789020 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.789026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:40.789241 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:40.789879 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:40.789897 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:40.793517 (+ 3620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":338,"threads_started":1}
W20260807 06:49:40.793964 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:40.784259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:43193 (local address 127.27.54.126:40025)
0807 06:49:40.784495 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:40.784501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:40.784984 (+ 483us) server_negotiation.cc:407] Connection header received
0807 06:49:40.785084 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:40.785088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:40.785164 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:40.785318 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:40.786292 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.787113 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.788434 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:40.788846 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:40.789000 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:40.793742 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:43193: BlockingRecv error: recv got EOF from 127.27.54.66:43193 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 07 06:49:41 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:41 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:41.242465 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:41.232907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:41107)
0807 06:49:41.233541 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:41.233569 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:41.233698 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:41.234105 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:41.234110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:41.234697 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:41.234885 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:41.234894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.235923 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.235926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:41.236849 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:41.236860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.237706 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.237711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:41.237846 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:41.238470 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:41.238488 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:41.242181 (+ 3693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":395,"threads_started":1}
W20260807 06:49:41.242615 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:41.232988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:41107 (local address 127.27.54.126:40025)
0807 06:49:41.233191 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:41.233199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:41.233673 (+ 474us) server_negotiation.cc:407] Connection header received
0807 06:49:41.233833 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:41.233851 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:41.233939 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:41.234028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:41.235018 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.235789 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.236989 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.237564 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.237663 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:41.242377 (+ 4714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:41107: BlockingRecv error: recv got EOF from 127.27.54.65:41107 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 06:49:41 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:41 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:41.704542 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:41.695080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:51565)
0807 06:49:41.695660 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:41.695685 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:41.695829 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:41.696201 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:41.696208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:41.696706 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:41.696865 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:41.696874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.697948 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.697953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:41.698951 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:41.698964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.699680 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.699686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:41.699874 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:41.700516 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:41.700533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:41.704184 (+ 3651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
W20260807 06:49:41.704684 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:41.695154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:51565 (local address 127.27.54.126:40025)
0807 06:49:41.695367 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:41.695375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:41.695760 (+ 385us) server_negotiation.cc:407] Connection header received
0807 06:49:41.695974 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:41.695984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:41.696050 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:41.696154 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:41.697022 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.697785 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.699102 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.699524 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.699663 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:41.704445 (+ 4782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:51565: BlockingRecv error: recv got EOF from 127.27.54.67:51565 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 06:49:41 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:41 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:41.804023 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:41.794947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:48299)
0807 06:49:41.795551 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:41.795582 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:41.795704 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:41.796018 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:41.796021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:41.796441 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:41.796620 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:41.796628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.797630 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.797633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:41.798584 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:41.798595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.799307 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.799311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:41.799445 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:41.800093 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:41.800110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:41.803777 (+ 3667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":371,"threads_started":1}
W20260807 06:49:41.804302 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:41.795017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:48299 (local address 127.27.54.126:40025)
0807 06:49:41.795257 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:41.795263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:41.795653 (+ 390us) server_negotiation.cc:407] Connection header received
0807 06:49:41.795803 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:41.795807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:41.795883 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:41.795968 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:41.796757 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.797497 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.798748 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:41.799161 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:41.799257 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:41.804040 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:48299: BlockingRecv error: recv got EOF from 127.27.54.66:48299 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 06:49:42 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:42 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:42.253680 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:42.243768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:58947)
0807 06:49:42.244387 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:42.244404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:42.244577 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:42.244940 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:42.244945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:42.245435 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:42.245607 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:42.245616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.246744 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.246757 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:42.247730 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:42.247742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.248509 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.248516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:42.248694 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:42.249374 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:42.249392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:42.253422 (+ 4030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":273,"threads_started":1}
W20260807 06:49:42.253924 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:42.243824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:58947 (local address 127.27.54.126:40025)
0807 06:49:42.244016 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:42.244021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:42.244501 (+ 480us) server_negotiation.cc:407] Connection header received
0807 06:49:42.244719 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:42.244730 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:42.244799 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:42.244906 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:42.245764 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.246556 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.247894 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.248334 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.248471 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:42.253682 (+ 5211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:58947: BlockingRecv error: recv got EOF from 127.27.54.65:58947 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 06:49:42 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:42 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:42.716312 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:42.705947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:50015)
0807 06:49:42.706540 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:42.706569 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:42.706788 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:42.707261 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:42.707267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:42.707942 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:42.708185 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:42.708196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.709625 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.709629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:42.710690 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:42.710705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.711419 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.711423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:42.711554 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:42.712193 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:42.712212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:42.716069 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":338,"threads_started":1}
W20260807 06:49:42.716581 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:42.706212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:50015 (local address 127.27.54.126:40025)
0807 06:49:42.706406 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:42.706411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:42.706688 (+ 277us) server_negotiation.cc:407] Connection header received
0807 06:49:42.706943 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:42.706949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:42.707044 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:42.707157 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:42.708378 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.709458 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.710828 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.711249 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.711362 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:42.716313 (+ 4951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:50015: BlockingRecv error: recv got EOF from 127.27.54.67:50015 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 06:49:42 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:42 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:42.815508 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:42.805238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:58605)
0807 06:49:42.805937 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:42.805961 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:42.806102 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:42.806611 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:42.806617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:42.807217 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:42.807481 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:42.807498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.808924 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.808928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:42.809911 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:42.809924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.810625 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.810630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:42.810851 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:42.811563 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:42.811582 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:42.815239 (+ 3657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":343,"threads_started":1}
W20260807 06:49:42.815713 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:42.805391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:58605 (local address 127.27.54.126:40025)
0807 06:49:42.805618 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:42.805623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:42.806068 (+ 445us) server_negotiation.cc:407] Connection header received
0807 06:49:42.806247 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:42.806252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:42.806340 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:42.806468 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:42.807662 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.808782 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.810053 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:42.810460 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:42.810558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:42.815475 (+ 4917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:58605: BlockingRecv error: recv got EOF from 127.27.54.66:58605 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 07 06:49:43 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:43 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:43.264386 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:43.254892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:59291)
0807 06:49:43.255446 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:43.255462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:43.255631 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:43.256024 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:43.256029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:43.256493 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:43.256670 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:43.256680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.257721 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.257728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:43.258761 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:43.258774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.259554 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.259562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:43.259728 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:43.260362 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:43.260380 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:43.264121 (+ 3741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
W20260807 06:49:43.264599 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:43.254992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:59291 (local address 127.27.54.126:40025)
0807 06:49:43.255235 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:43.255244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:43.255558 (+ 314us) server_negotiation.cc:407] Connection header received
0807 06:49:43.255800 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:43.255807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:43.255878 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:43.255985 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:43.256827 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.257576 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.258903 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.259355 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.259490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:43.264358 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:59291: BlockingRecv error: recv got EOF from 127.27.54.65:59291 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 06:49:43 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:43 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:43.732069 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:43.717417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:37185)
0807 06:49:43.718137 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:43.718154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:43.718335 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:43.718744 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:43.718749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:43.719260 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:43.719513 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:43.719525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.720670 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.720675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:43.721732 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:43.721745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.722522 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.722528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:43.722717 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:43.727897 (+ 5180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:43.727916 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:43.731682 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":328,"threads_started":1}
W20260807 06:49:43.732156 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:43.717639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:37185 (local address 127.27.54.126:40025)
0807 06:49:43.717949 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:43.717955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:43.718262 (+ 307us) server_negotiation.cc:407] Connection header received
0807 06:49:43.718482 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:43.718492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:43.718558 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:43.718700 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:43.719674 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.720521 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.721896 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.722356 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.722499 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:43.731909 (+ 9410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:37185: BlockingRecv error: recv got EOF from 127.27.54.67:37185 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 07 06:49:43 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:43 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:43.826713 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:43.816870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:58281)
0807 06:49:43.817465 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:43.817493 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:43.817679 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:43.818041 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:43.818045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:43.818510 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:43.818766 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:43.818778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.819990 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.819996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:43.820952 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:43.820964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.821756 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.821762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:43.821942 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:43.822751 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:43.822769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:43.826394 (+ 3625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":397,"threads_started":1}
W20260807 06:49:43.827003 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:43.817046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:58281 (local address 127.27.54.126:40025)
0807 06:49:43.817265 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:43.817272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:43.817606 (+ 334us) server_negotiation.cc:407] Connection header received
0807 06:49:43.817818 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:43.817823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:43.817894 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:43.818003 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:43.818920 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.819836 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.821106 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:43.821551 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:43.821695 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:43.826631 (+ 4936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:58281: BlockingRecv error: recv got EOF from 127.27.54.66:58281 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 06:49:44 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:44 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:44.275432 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:44.265717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:38181)
0807 06:49:44.266337 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:44.266354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:44.266527 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:44.266962 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:44.266967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:44.267440 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:44.267588 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:44.267596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.268730 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.268739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:44.269867 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:44.269880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.270705 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.270712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:44.270895 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:44.271495 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:44.271512 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:44.275078 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
W20260807 06:49:44.275593 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:44.265825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:38181 (local address 127.27.54.126:40025)
0807 06:49:44.266114 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:44.266120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:44.266466 (+ 346us) server_negotiation.cc:407] Connection header received
0807 06:49:44.266724 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:44.266729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:44.266820 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:44.266931 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:44.267773 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.268585 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.270009 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.270493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.270701 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:44.275321 (+ 4620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:38181: BlockingRecv error: recv got EOF from 127.27.54.65:38181 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 07 06:49:44 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:44 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:44.744482 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:44.733427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:44803)
0807 06:49:44.734072 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:44.734091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:44.734311 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:44.734825 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:44.734830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:44.735457 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:44.735746 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:44.735765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.737042 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.737050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:44.738280 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:44.738295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.739097 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.739104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:44.739262 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:44.739973 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:44.739993 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:44.744210 (+ 4217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":343,"threads_started":1}
W20260807 06:49:44.744762 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:44.733526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:44803 (local address 127.27.54.126:40025)
0807 06:49:44.733827 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:44.733836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:44.734228 (+ 392us) server_negotiation.cc:407] Connection header received
0807 06:49:44.734460 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:44.734473 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:44.734544 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:44.734792 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:44.735917 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.736873 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.738423 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.738885 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.738986 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:44.744459 (+ 5473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:44803: BlockingRecv error: recv got EOF from 127.27.54.67:44803 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 07 06:49:44 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:44 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:44.838231 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:44.827968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:45923)
0807 06:49:44.828554 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:44.828577 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:44.828700 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:44.829018 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:44.829021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:44.829472 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:44.829625 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:44.829634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.830716 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.830721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:44.831745 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:44.831758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.832503 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.832508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:44.832646 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:44.833297 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:44.833314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:44.837854 (+ 4540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":315,"threads_started":1}
W20260807 06:49:44.838316 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:44.828019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:45923 (local address 127.27.54.126:40025)
0807 06:49:44.828334 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:44.828338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:44.828664 (+ 326us) server_negotiation.cc:407] Connection header received
0807 06:49:44.828806 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:44.828810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:44.828874 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:44.828964 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:44.829798 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.830538 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.831905 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:44.832352 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:44.832456 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:44.838048 (+ 5592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:45923: BlockingRecv error: recv got EOF from 127.27.54.66:45923 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Aug 07 06:49:45 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:45 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:45.286798 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:45.276609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:57559)
0807 06:49:45.277274 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:45.277303 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:45.277430 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:45.277733 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:45.277737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:45.278139 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:45.278331 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:45.278341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.279440 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.279445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:45.280487 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:45.280501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.281239 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.281244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:45.281368 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:45.282186 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:45.282210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:45.286455 (+ 4245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":387,"threads_started":1}
W20260807 06:49:45.286958 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:45.276734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:57559 (local address 127.27.54.126:40025)
0807 06:49:45.276930 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:45.276934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:45.277386 (+ 452us) server_negotiation.cc:407] Connection header received
0807 06:49:45.277525 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:45.277528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:45.277593 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:45.277677 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:45.278473 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.279278 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.280633 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.281058 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.281154 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:45.286702 (+ 5548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:57559: BlockingRecv error: recv got EOF from 127.27.54.65:57559 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 06:49:45 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:45 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:45.755227 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:45.745610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:36409)
0807 06:49:45.746371 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:45.746388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:45.746559 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:45.746994 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:45.747001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:45.747529 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:45.747682 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:45.747691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.748696 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.748700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:45.749635 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:45.749646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.750408 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.750416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:45.750568 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:45.751244 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:45.751261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:45.754984 (+ 3723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":322,"threads_started":1}
W20260807 06:49:45.755384 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:45.745683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:36409 (local address 127.27.54.126:40025)
0807 06:49:45.745962 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:45.745970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:45.746496 (+ 526us) server_negotiation.cc:407] Connection header received
0807 06:49:45.746727 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:45.746733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:45.746821 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:45.746931 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:45.747827 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.748570 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.749800 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.750228 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.750340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:45.755186 (+ 4846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:36409: BlockingRecv error: recv got EOF from 127.27.54.67:36409 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 06:49:45 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:45 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:45.849972 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:45.839486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:51287)
0807 06:49:45.840115 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:45.840140 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:45.840271 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:45.840590 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:45.840594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:45.841040 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:45.841213 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:45.841222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.842430 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.842434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:45.843633 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:45.843648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.844387 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.844392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:45.844513 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:45.845186 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:45.845207 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:45.849678 (+ 4471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":353,"threads_started":1}
W20260807 06:49:45.850270 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:45.839579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:51287 (local address 127.27.54.126:40025)
0807 06:49:45.839786 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:45.839791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:45.840262 (+ 471us) server_negotiation.cc:407] Connection header received
0807 06:49:45.840377 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:45.840380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:45.840450 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:45.840548 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:45.841373 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.842234 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.843789 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:45.844237 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:45.844344 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:45.849938 (+ 5594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:51287: BlockingRecv error: recv got EOF from 127.27.54.66:51287 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 06:49:46 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:46 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:46.298049 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:46.288042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:37345)
0807 06:49:46.288646 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:46.288664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:46.288797 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:46.289169 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:46.289173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:46.289761 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:46.290051 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:46.290063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.291479 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.291485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:46.292475 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:46.292489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.293145 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.293149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:46.293296 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:46.294012 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:46.294030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:46.297793 (+ 3763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":353,"threads_started":1}
W20260807 06:49:46.298337 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:46.288293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:37345 (local address 127.27.54.126:40025)
0807 06:49:46.288560 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:46.288564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:46.288753 (+ 189us) server_negotiation.cc:407] Connection header received
0807 06:49:46.288918 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:46.288922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:46.288990 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:46.289089 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:46.290211 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.291281 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.292619 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.293005 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.293106 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:46.298044 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:37345: BlockingRecv error: recv got EOF from 127.27.54.65:37345 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Aug 07 06:49:46 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:46 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:46.770545 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:46.756181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:42937)
0807 06:49:46.756725 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:46.756743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:46.756866 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:46.757187 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:46.757191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:46.757612 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:46.757779 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:46.757787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.759011 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.759015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:46.760025 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:46.760037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.760795 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.760800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:46.760916 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:46.766255 (+ 5339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:46.766277 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:46.770246 (+ 3969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":324,"threads_started":1}
W20260807 06:49:46.770831 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:46.756276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:42937 (local address 127.27.54.126:40025)
0807 06:49:46.756535 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:46.756542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:46.756829 (+ 287us) server_negotiation.cc:407] Connection header received
0807 06:49:46.756976 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:46.756980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:46.757048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:46.757150 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:46.757947 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.758870 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.760171 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.760639 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.760744 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:46.770529 (+ 9785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:42937: BlockingRecv error: recv got EOF from 127.27.54.67:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 06:49:46 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:46 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:46.861171 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:46.851150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:50287)
0807 06:49:46.851718 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:46.851744 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:46.851882 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:46.852243 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:46.852247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:46.852687 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:46.852903 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:46.852912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.854012 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.854018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:46.855394 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:46.855415 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.856416 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.856420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:46.856553 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:46.857218 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:46.857236 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:46.860915 (+ 3679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":327,"threads_started":1}
W20260807 06:49:46.861591 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:46.851282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:50287 (local address 127.27.54.126:40025)
0807 06:49:46.851582 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:46.851587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:46.851854 (+ 267us) server_negotiation.cc:407] Connection header received
0807 06:49:46.851996 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:46.852000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:46.852085 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:46.852198 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:46.853061 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.853857 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.855568 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:46.856242 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:46.856359 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:46.861157 (+ 4798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:50287: BlockingRecv error: recv got EOF from 127.27.54.66:50287 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Aug 07 06:49:47 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:47 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:47.309598 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:47.299319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:41433)
0807 06:49:47.299962 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:47.299984 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:47.300149 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:47.300551 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:47.300555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:47.301010 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:47.301198 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:47.301207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.302326 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.302330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:47.303747 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:47.303764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.304628 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.304635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:47.304849 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:47.305530 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:47.305550 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:47.309344 (+ 3794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":351,"threads_started":1}
W20260807 06:49:47.309836 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:47.299429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:41433 (local address 127.27.54.126:40025)
0807 06:49:47.299701 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:47.299708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:47.300087 (+ 379us) server_negotiation.cc:407] Connection header received
0807 06:49:47.300295 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:47.300299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:47.300368 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:47.300458 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:47.301380 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.302160 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.303931 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.304413 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.304593 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:47.309557 (+ 4964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:41433: BlockingRecv error: recv got EOF from 127.27.54.65:41433 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 06:49:47 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:47 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:47.780961 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:47.771582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:46619)
0807 06:49:47.772141 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:47.772159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:47.772288 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:47.772617 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:47.772621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:47.773043 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:47.773241 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:47.773250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.774324 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.774328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:47.775471 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:47.775484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.776442 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.776447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:47.776578 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:47.777192 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:47.777209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:47.780677 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":321,"threads_started":1}
W20260807 06:49:47.781147 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:47.771716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:46619 (local address 127.27.54.126:40025)
0807 06:49:47.771897 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:47.771901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:47.772239 (+ 338us) server_negotiation.cc:407] Connection header received
0807 06:49:47.772393 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:47.772397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:47.772461 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:47.772559 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:47.773397 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.774189 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.775650 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.776266 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.776411 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:47.780931 (+ 4520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:46619: BlockingRecv error: recv got EOF from 127.27.54.67:46619 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 06:49:47 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:47 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:47.873090 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:47.862363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:51417)
0807 06:49:47.863137 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:47.863161 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:47.863323 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:47.864036 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:47.864044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:47.864741 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:47.864947 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:47.864957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.866056 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.866062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:47.867083 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:47.867097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.867839 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.867843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:47.867977 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:47.868776 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:47.868796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:47.872824 (+ 4028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":422,"threads_started":1}
W20260807 06:49:47.873317 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:47.862510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:51417 (local address 127.27.54.126:40025)
0807 06:49:47.862861 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:47.862869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:47.863250 (+ 381us) server_negotiation.cc:407] Connection header received
0807 06:49:47.863511 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:47.863516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:47.863607 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:47.863707 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:47.865086 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.865910 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.867203 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:47.867674 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:47.867770 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:47.873045 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:51417: BlockingRecv error: recv got EOF from 127.27.54.66:51417 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 07 06:49:48 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:48 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:48.320559 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:48.310838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:52381)
0807 06:49:48.311425 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:48.311443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:48.311618 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:48.312012 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:48.312017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:48.312492 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:48.312653 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:48.312664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.313690 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.313697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:48.314631 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:48.314642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.315428 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.315434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:48.315612 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:48.316341 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:48.316360 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:48.320323 (+ 3963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":341,"threads_started":1}
W20260807 06:49:48.320843 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:48.310939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:52381 (local address 127.27.54.126:40025)
0807 06:49:48.311227 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:48.311235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:48.311543 (+ 308us) server_negotiation.cc:407] Connection header received
0807 06:49:48.311794 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:48.311798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:48.311865 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:48.311974 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:48.312808 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.313535 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.314818 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.315266 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.315401 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:48.320521 (+ 5120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:52381: BlockingRecv error: recv got EOF from 127.27.54.65:52381 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Aug 07 06:49:48 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:48 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:48.797487 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:48.782192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:47059)
0807 06:49:48.783089 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:48.783106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:48.783231 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:48.783520 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:48.783523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:48.783986 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:48.784160 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:48.784169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.785432 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.785436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:48.786503 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:48.786517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.787393 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.787398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:48.787534 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:48.792723 (+ 5189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:48.792745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:48.797229 (+ 4484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":524,"threads_started":1}
W20260807 06:49:48.797742 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:48.782255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:47059 (local address 127.27.54.126:40025)
0807 06:49:48.782549 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:48.782559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:48.783206 (+ 647us) server_negotiation.cc:407] Connection header received
0807 06:49:48.783320 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:48.783324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:48.783393 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:48.783484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:48.784293 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.785265 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.786766 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.787211 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.787322 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:48.797452 (+ 10130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:47059: BlockingRecv error: recv got EOF from 127.27.54.67:47059 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 06:49:48 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:48 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:48.884321 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:48.874316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:38511)
0807 06:49:48.874919 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:48.874953 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:48.875081 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:48.875416 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:48.875419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:48.875852 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:48.876021 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:48.876029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.877211 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.877215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:48.878207 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:48.878220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.879146 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.879151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:48.879297 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:48.880002 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:48.880022 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:48.884067 (+ 4045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":365,"threads_started":1}
W20260807 06:49:48.884594 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:48.874587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:38511 (local address 127.27.54.126:40025)
0807 06:49:48.874857 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:48.874862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:48.875034 (+ 172us) server_negotiation.cc:407] Connection header received
0807 06:49:48.875174 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:48.875178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:48.875262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:48.875362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:48.876239 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.877057 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.878373 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:48.878956 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:48.879073 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:48.884309 (+ 5236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:38511: BlockingRecv error: recv got EOF from 127.27.54.66:38511 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 07 06:49:49 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:49 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:49.332512 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:49.322086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:56343)
0807 06:49:49.322858 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:49.322877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:49.323063 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:49.323456 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:49.323462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:49.323924 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:49.324125 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:49.324135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.325262 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.325273 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:49.326263 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:49.326284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.327308 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.327325 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:49.327542 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:49.328390 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:49.328420 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:49.332235 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":387,"threads_started":1}
W20260807 06:49:49.332804 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:49.322276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:56343 (local address 127.27.54.126:40025)
0807 06:49:49.322570 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:49.322576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:49.322982 (+ 406us) server_negotiation.cc:407] Connection header received
0807 06:49:49.323214 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:49.323219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:49.323299 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:49.323411 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:49.324292 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.325099 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.326470 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.327062 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.327244 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:49.332493 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:56343: BlockingRecv error: recv got EOF from 127.27.54.65:56343 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 07 06:49:49 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:49 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.67@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:49.808983 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:49.798865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.67:36981)
0807 06:49:49.799471 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:49.799490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:49.799669 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:49.800085 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:49.800090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:49.800583 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:49.800752 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:49.800765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.801895 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.801901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:49.803063 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:49.803079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.803913 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.803920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:49.804101 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:49.804942 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:49.804961 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:49.808710 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":337,"threads_started":1}
W20260807 06:49:49.809307 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:49.798975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:36981 (local address 127.27.54.126:40025)
0807 06:49:49.799343 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:49.799349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:49.799589 (+ 240us) server_negotiation.cc:407] Connection header received
0807 06:49:49.799853 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:49.799857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:49.799935 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:49.800044 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:49.800924 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.801724 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.803237 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.803749 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.803893 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:49.808998 (+ 5105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.67:36981: BlockingRecv error: recv got EOF from 127.27.54.67:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Aug 07 06:49:49 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:49 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.66@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:49.895704 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:49.885722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.66:55771)
0807 06:49:49.886256 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:49.886276 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:49.886465 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:49.886867 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:49.886872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:49.887335 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:49.887538 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:49.887548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.888601 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.888607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:49.889576 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:49.889588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.890422 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.890428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:49.890633 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:49.891500 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:49.891518 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:49.895446 (+ 3928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
W20260807 06:49:49.896013 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:49.885824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:55771 (local address 127.27.54.126:40025)
0807 06:49:49.886103 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:49.886110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:49.886384 (+ 274us) server_negotiation.cc:407] Connection header received
0807 06:49:49.886602 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:49.886606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:49.886722 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:49.886828 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:49.887696 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.888443 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.889729 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:49.890251 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:49.890399 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:49.895677 (+ 5278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.66:55771: BlockingRecv error: recv got EOF from 127.27.54.66:55771 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 07 06:49:50 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 06:49:50 dist-test-slave-3kk6 krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.54.65@KRBTEST.COM for oryx/127.27.54.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 06:49:50.344177 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:50.334203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40025 (local address 127.27.54.65:44005)
0807 06:49:50.334863 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:50.334881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:50.335063 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:50.335459 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:50.335466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:50.336067 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:50.336281 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:50.336293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.337310 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.337314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:50.338281 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:50.338293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.339130 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.339136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:50.339289 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:50.340022 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:50.340041 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:50.343912 (+ 3871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.54.126:40025: Server oryx/127.27.54.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":365,"threads_started":1}
W20260807 06:49:50.344499 9111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:50.334287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:44005 (local address 127.27.54.126:40025)
0807 06:49:50.334484 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:50.334490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:50.334985 (+ 495us) server_negotiation.cc:407] Connection header received
0807 06:49:50.335206 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:50.335211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:50.335286 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:50.335403 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:50.336430 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.337181 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.338436 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.338957 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.339070 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:50.344191 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.54.65:44005: BlockingRecv error: recv got EOF from 127.27.54.65:44005 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 06:49:50.473202 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8474
I20260807 06:49:50.487272 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:50.474525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.27.54.65:45851)
0807 06:49:50.475251 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:50.475259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:50.475291 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:49:50.475358 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:50.475364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:50.475679 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:50.475844 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:50.477526 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.478690 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.479940 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.480518 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.480620 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:50.483434 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:50.483445 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:50.483461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:50.483519 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:50.486580 (+ 3061us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.54.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 06:49:50.486967 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40498: Request ticket server kudu/127.27.54.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":382,"threads_started":1}
W20260807 06:49:50.487396 9187 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40498: Request ticket server kudu/127.27.54.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 06:49:50.487488 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8474: Not authorized: Client connection negotiation failed: client connection to 127.27.54.65:45851: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.54.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 06:49:50.487715 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 8474
I20260807 06:49:50.516103 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8707
I20260807 06:49:50.530099 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:50.517671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.27.54.66:37405)
0807 06:49:50.518344 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:50.518355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:50.518391 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:49:50.518472 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:50.518482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:50.518864 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:50.519100 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:50.520542 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.521602 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.523086 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.523669 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.523791 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:50.526536 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:50.526545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:50.526560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:50.526617 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:50.529530 (+ 2913us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.54.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 06:49:50.529852 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40292: Request ticket server kudu/127.27.54.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":286,"threads_started":1}
W20260807 06:49:50.530221 9189 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40292: Request ticket server kudu/127.27.54.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 06:49:50.530254 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8707: Not authorized: Client connection negotiation failed: client connection to 127.27.54.66:37405: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.54.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 06:49:50.530438 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 8707
I20260807 06:49:50.555256 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8902
I20260807 06:49:50.569613 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:50.557026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.27.54.67:39355)
0807 06:49:50.557708 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:50.557718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:50.557754 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:49:50.557838 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:50.557859 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:50.558197 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:50.558434 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:50.559973 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.561073 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.562276 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:50.562993 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:50.563106 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:50.565875 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:50.565885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:50.565902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:50.565949 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:50.568904 (+ 2955us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.54.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 06:49:50.569296 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39944: Request ticket server kudu/127.27.54.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":305,"threads_started":1}
W20260807 06:49:50.569797 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8902: Not authorized: Client connection negotiation failed: client connection to 127.27.54.67:39355: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.54.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 06:49:50.569897 9191 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39944: Request ticket server kudu/127.27.54.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 06:49:50.569967 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 8902
I20260807 06:49:50.595568 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 8359
I20260807 06:49:50.597126 8461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:50.720958 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 8359
2026-08-07T06:49:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22511 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T06:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:50Z Disabled control of system clock
I20260807 06:49:50.830163 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38821
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 06:49:51.200479 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:51.200850 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:51.200991 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:51.208689 9197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:51.208854 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:51.208897 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:51.209020 9197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:51.209087 9197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:51.218861 9197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:38821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:51.220345 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:51.222458 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:51.235316 9208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:51.235455 9206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:51.235455 9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:51.235858 9197 server_base.cc:1034] running on GCE node
I20260807 06:49:51.237013 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:51.238574 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:51.239809 9197 hybrid_clock.cc:648] HybridClock initialized: now 1786085391239766 us; error 34 us; skew 500 ppm
I20260807 06:49:51.240207 9197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:51.242832 9197 webserver.cc:460] Webserver started at http://127.27.54.126:40981/ using document root <none> and password file <none>
I20260807 06:49:51.243597 9197 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:51.243738 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:51.244117 9197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:51.247342 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "e5ab2e53748c4c1cbfc7282fab860417"
format_stamp: "Formatted at 2026-08-07 06:49:51 on dist-test-slave-3kk6"
server_key: "cde74cf2398029d94fe6acf2340ea45c"
server_key_iv: "5a9ec82e89048f001057978e091617b3"
server_key_version: "encryptionkey@0"
I20260807 06:49:51.248219 9197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "e5ab2e53748c4c1cbfc7282fab860417"
format_stamp: "Formatted at 2026-08-07 06:49:51 on dist-test-slave-3kk6"
server_key: "cde74cf2398029d94fe6acf2340ea45c"
server_key_iv: "5a9ec82e89048f001057978e091617b3"
server_key_version: "encryptionkey@0"
I20260807 06:49:51.253865 9197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:49:51.257778 9216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:51.259322 9197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:51.259500 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "e5ab2e53748c4c1cbfc7282fab860417"
format_stamp: "Formatted at 2026-08-07 06:49:51 on dist-test-slave-3kk6"
server_key: "cde74cf2398029d94fe6acf2340ea45c"
server_key_iv: "5a9ec82e89048f001057978e091617b3"
server_key_version: "encryptionkey@0"
I20260807 06:49:51.259764 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:51.280983 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:51.281968 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:51.282305 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:51.302482 9197 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:38821
I20260807 06:49:51.302485 9298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:38821 every 8 connection(s)
I20260807 06:49:51.304005 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:51.308089 9299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:51.314337 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 9197
I20260807 06:49:51.314594 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:51.315290 27865 external_mini_cluster.cc:1428] Setting key e7cd66d813aa03f365cc86d81e248e76
I20260807 06:49:51.319283 9299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417: Bootstrap starting.
I20260807 06:49:51.322849 9299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:51.324123 9299 log.cc:824] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:51.327698 9299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417: No bootstrap required, opened a new log
I20260807 06:49:51.330173 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:51.317568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.27.54.126:38821)
0807 06:49:51.318695 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:51.318714 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:51.318763 (+ 49us) server_negotiation.cc:407] Connection header received
0807 06:49:51.320575 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:51.320598 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:51.321068 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:51.321413 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:51.322198 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:51.323664 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:51.325455 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:51.326164 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:51.326315 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:51.327719 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:51.327731 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:51.327746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:51.328355 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:51.328440 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:51.328466 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:51.328560 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:51.328793 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 06:49:51.329152 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":412,"threads_started":1}
I20260807 06:49:51.336160 9299 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab2e53748c4c1cbfc7282fab860417" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38821 } }
I20260807 06:49:51.336583 9299 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:51.336670 9299 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ab2e53748c4c1cbfc7282fab860417, State: Initialized, Role: FOLLOWER
I20260807 06:49:51.337193 9299 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab2e53748c4c1cbfc7282fab860417" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38821 } }
I20260807 06:49:51.337393 9299 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:51.337497 9299 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:51.337656 9299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:51.340461 9299 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab2e53748c4c1cbfc7282fab860417" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38821 } }
I20260807 06:49:51.340967 9299 leader_election.cc:302] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ab2e53748c4c1cbfc7282fab860417; no voters:
I20260807 06:49:51.341588 9299 leader_election.cc:288] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:51.341768 9307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:51.342516 9307 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [term 1 LEADER]: Becoming Leader. State: Replica: e5ab2e53748c4c1cbfc7282fab860417, State: Running, Role: LEADER
I20260807 06:49:51.343319 9307 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab2e53748c4c1cbfc7282fab860417" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38821 } }
I20260807 06:49:51.344022 9299 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:51.346459 9308 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ab2e53748c4c1cbfc7282fab860417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab2e53748c4c1cbfc7282fab860417" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38821 } } }
I20260807 06:49:51.346963 9308 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:51.346436 9309 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ab2e53748c4c1cbfc7282fab860417. Latest consensus state: current_term: 1 leader_uuid: "e5ab2e53748c4c1cbfc7282fab860417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ab2e53748c4c1cbfc7282fab860417" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38821 } } }
I20260807 06:49:51.347355 9309 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:51.349360 9315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:51.354429 9315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:51.364356 9315 catalog_manager.cc:1347] Generated new cluster ID: 3f0e94a9106f4d33a601580f355c9d46
I20260807 06:49:51.364602 9315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:51.379817 9315 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:51.381889 9315 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:51.392105 9315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ab2e53748c4c1cbfc7282fab860417: Generated new TSK 0
I20260807 06:49:51.393088 9315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:49:51.407845 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38821
--builtin_ntp_servers=127.27.54.84:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 06:49:51.801019 9331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:51.801362 9331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:51.801453 9331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:51.808919 9331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:51.809057 9331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:51.809168 9331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:51.818804 9331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:51.820181 9331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:51.822098 9331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:51.836972 9336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:51.837021 9337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:51.837047 9343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:51.837965 9331 server_base.cc:1034] running on GCE node
I20260807 06:49:51.838723 9331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:51.839862 9331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:51.841099 9331 hybrid_clock.cc:648] HybridClock initialized: now 1786085391841028 us; error 75 us; skew 500 ppm
I20260807 06:49:51.841455 9331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:51.844319 9331 webserver.cc:460] Webserver started at http://127.27.54.65:38149/ using document root <none> and password file <none>
I20260807 06:49:51.845136 9331 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:51.845359 9331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:51.845711 9331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:51.848953 9331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "05707b0c9269404d8062e8f06e22f925"
format_stamp: "Formatted at 2026-08-07 06:49:51 on dist-test-slave-3kk6"
server_key: "04a207e0a12d475b2641a55cfa87c3d5"
server_key_iv: "b98c99803d9390a55991f8220ad56bf6"
server_key_version: "encryptionkey@0"
I20260807 06:49:51.849911 9331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "05707b0c9269404d8062e8f06e22f925"
format_stamp: "Formatted at 2026-08-07 06:49:51 on dist-test-slave-3kk6"
server_key: "04a207e0a12d475b2641a55cfa87c3d5"
server_key_iv: "b98c99803d9390a55991f8220ad56bf6"
server_key_version: "encryptionkey@0"
I20260807 06:49:51.855640 9331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 06:49:51.859357 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:51.861027 9331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:51.861253 9331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "05707b0c9269404d8062e8f06e22f925"
format_stamp: "Formatted at 2026-08-07 06:49:51 on dist-test-slave-3kk6"
server_key: "04a207e0a12d475b2641a55cfa87c3d5"
server_key_iv: "b98c99803d9390a55991f8220ad56bf6"
server_key_version: "encryptionkey@0"
I20260807 06:49:51.861527 9331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:51.881364 9331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:51.882356 9331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:51.882786 9331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:51.883905 9331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:51.885569 9331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:51.885731 9331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:51.885890 9331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:51.885980 9331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:51.922513 9331 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:35205
I20260807 06:49:51.922575 9519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:35205 every 8 connection(s)
I20260807 06:49:51.924065 9331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:51.927387 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 9331
I20260807 06:49:51.927629 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:51.928563 27865 external_mini_cluster.cc:1428] Setting key 2e882dca8b076d710c6b8f76d0ade9ff
I20260807 06:49:51.931602 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38821
--builtin_ntp_servers=127.27.54.84:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 06:49:51.938055 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:51.926918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:39413 (local address 127.27.54.126:38821)
0807 06:49:51.927399 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:51.927404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:51.928962 (+ 1558us) server_negotiation.cc:407] Connection header received
0807 06:49:51.931130 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:51.931135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:51.931228 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:51.931353 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:51.932638 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:51.933379 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:51.935088 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:51.935506 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:51.935644 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:51.937156 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:51.937161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:51.937164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:51.937185 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:51.937227 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:51.937239 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:51.937341 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:51.937581 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 06:49:51.937694 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260807 06:49:51.939254 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:51.927687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38821 (local address 127.27.54.65:39413)
0807 06:49:51.928783 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:51.928831 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:51.930838 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:51.931614 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:51.931623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:51.931774 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:51.932460 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:51.932479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:51.933560 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:51.933564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:51.934924 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:51.934944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:51.935751 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:51.935770 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:51.936000 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:51.936817 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:49:51.936820 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:51.936863 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:49:51.936946 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:51.937358 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:51.937377 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:51.937384 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:49:51.937570 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 06:49:51.938019 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":334,"threads_started":1}
I20260807 06:49:51.941083 9523 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38821
I20260807 06:49:51.941473 9523 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:51.942516 9523 heartbeater.cc:511] Master 127.27.54.126:38821 requested a full tablet report, sending...
I20260807 06:49:51.945097 9243 ts_manager.cc:194] Registered new tserver with Master: 05707b0c9269404d8062e8f06e22f925 (127.27.54.65:35205)
I20260807 06:49:51.946808 9243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.65:39413
W20260807 06:49:52.305980 9528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:52.306331 9528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:52.306501 9528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:52.314328 9528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:52.314507 9528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:52.314752 9528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:52.324615 9528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:52.326180 9528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:52.328179 9528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:52.342566 9533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:52.342589 9535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:52.342617 9538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:52.343395 9528 server_base.cc:1034] running on GCE node
I20260807 06:49:52.344141 9528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:52.345279 9528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:52.346546 9528 hybrid_clock.cc:648] HybridClock initialized: now 1786085392346490 us; error 54 us; skew 500 ppm
I20260807 06:49:52.346973 9528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:52.349732 9528 webserver.cc:460] Webserver started at http://127.27.54.66:34681/ using document root <none> and password file <none>
I20260807 06:49:52.350536 9528 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:52.350694 9528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:52.351065 9528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:52.354280 9528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "7afd4fcd7abd4d91a0f91cf3bfcff75c"
format_stamp: "Formatted at 2026-08-07 06:49:52 on dist-test-slave-3kk6"
server_key: "bb2d4caab6d1391002b13d00edd5e8e6"
server_key_iv: "b635120dcb4e0efee13c6dd99d28708b"
server_key_version: "encryptionkey@0"
I20260807 06:49:52.355259 9528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "7afd4fcd7abd4d91a0f91cf3bfcff75c"
format_stamp: "Formatted at 2026-08-07 06:49:52 on dist-test-slave-3kk6"
server_key: "bb2d4caab6d1391002b13d00edd5e8e6"
server_key_iv: "b635120dcb4e0efee13c6dd99d28708b"
server_key_version: "encryptionkey@0"
I20260807 06:49:52.361137 9528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:49:52.365312 9548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:52.367050 9528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:52.367259 9528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "7afd4fcd7abd4d91a0f91cf3bfcff75c"
format_stamp: "Formatted at 2026-08-07 06:49:52 on dist-test-slave-3kk6"
server_key: "bb2d4caab6d1391002b13d00edd5e8e6"
server_key_iv: "b635120dcb4e0efee13c6dd99d28708b"
server_key_version: "encryptionkey@0"
I20260807 06:49:52.367579 9528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:52.389704 9528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:52.390784 9528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:52.391144 9528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:52.392382 9528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:52.394119 9528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:52.394224 9528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:52.394316 9528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:52.394372 9528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:52.431588 9528 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:38809
I20260807 06:49:52.431646 9715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:38809 every 8 connection(s)
I20260807 06:49:52.433285 9528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:52.438555 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 9528
I20260807 06:49:52.438799 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:52.439270 27865 external_mini_cluster.cc:1428] Setting key 910766809cfb133a289b172ac7ffc2cc
I20260807 06:49:52.442548 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38821
--builtin_ntp_servers=127.27.54.84:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 06:49:52.449229 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:52.436352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:56059 (local address 127.27.54.126:38821)
0807 06:49:52.436577 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:52.436582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:52.438145 (+ 1563us) server_negotiation.cc:407] Connection header received
0807 06:49:52.440364 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:52.440369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:52.440476 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:52.440597 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:52.441911 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.442726 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.445632 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.446111 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.446248 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:52.448312 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:52.448324 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:52.448327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:52.448354 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:52.448390 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:52.448403 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:52.448494 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:52.448790 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 06:49:52.448943 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 06:49:52.450959 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:52.436932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38821 (local address 127.27.54.66:56059)
0807 06:49:52.437945 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:52.437996 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:52.440013 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:52.440859 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:52.440869 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:52.440967 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:52.441733 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:52.441755 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.442935 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.442943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:52.445393 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:52.445418 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.446415 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.446426 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:52.446852 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:52.447824 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:49:52.447830 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:52.447895 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:49:52.448037 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:52.448516 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:52.448530 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:52.448542 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:49:52.448765 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 06:49:52.449296 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260807 06:49:52.452854 9716 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38821
I20260807 06:49:52.453300 9716 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:52.454494 9716 heartbeater.cc:511] Master 127.27.54.126:38821 requested a full tablet report, sending...
I20260807 06:49:52.456305 9243 ts_manager.cc:194] Registered new tserver with Master: 7afd4fcd7abd4d91a0f91cf3bfcff75c (127.27.54.66:38809)
I20260807 06:49:52.457187 9243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.66:56059
W20260807 06:49:52.816402 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:52.816762 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:52.816861 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:52.824517 9721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:52.824669 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:52.824786 9721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:52.834347 9721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:52.835837 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:52.837662 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:52.851822 9727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:52.851851 9732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:52.851775 9730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:52.852809 9721 server_base.cc:1034] running on GCE node
I20260807 06:49:52.853415 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:52.854368 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:52.855594 9721 hybrid_clock.cc:648] HybridClock initialized: now 1786085392855509 us; error 55 us; skew 500 ppm
I20260807 06:49:52.855935 9721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:52.858301 9721 webserver.cc:460] Webserver started at http://127.27.54.67:46563/ using document root <none> and password file <none>
I20260807 06:49:52.859270 9721 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:52.859421 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:52.859757 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:52.862881 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "76aa36f2b59047f8911a18ea17e645ac"
format_stamp: "Formatted at 2026-08-07 06:49:52 on dist-test-slave-3kk6"
server_key: "f244f2406090ebb76c8439f99946e787"
server_key_iv: "e777e7e1fc79f6ea25a549e707583b37"
server_key_version: "encryptionkey@0"
I20260807 06:49:52.863783 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "76aa36f2b59047f8911a18ea17e645ac"
format_stamp: "Formatted at 2026-08-07 06:49:52 on dist-test-slave-3kk6"
server_key: "f244f2406090ebb76c8439f99946e787"
server_key_iv: "e777e7e1fc79f6ea25a549e707583b37"
server_key_version: "encryptionkey@0"
I20260807 06:49:52.869620 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:49:52.873597 9742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:52.875073 9721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:49:52.875263 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "76aa36f2b59047f8911a18ea17e645ac"
format_stamp: "Formatted at 2026-08-07 06:49:52 on dist-test-slave-3kk6"
server_key: "f244f2406090ebb76c8439f99946e787"
server_key_iv: "e777e7e1fc79f6ea25a549e707583b37"
server_key_version: "encryptionkey@0"
I20260807 06:49:52.875562 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:52.898133 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:52.899170 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:52.899515 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:52.900617 9721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:52.902154 9721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:52.902279 9721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:52.902403 9721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:52.902490 9721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:52.936158 9721 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:35531
I20260807 06:49:52.936247 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:35531 every 8 connection(s)
I20260807 06:49:52.937737 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:52.940229 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 9721
I20260807 06:49:52.940455 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:49:52.941047 27865 external_mini_cluster.cc:1428] Setting key d86ed86a4abac19d46ae13d3b36ccdad
I20260807 06:49:52.950857 9523 heartbeater.cc:503] Master 127.27.54.126:38821 was elected leader, sending a full tablet report...
I20260807 06:49:52.951135 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:52.940747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:33127 (local address 127.27.54.126:38821)
0807 06:49:52.940955 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:52.940959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:52.942753 (+ 1794us) server_negotiation.cc:407] Connection header received
0807 06:49:52.944687 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:52.944690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:52.944765 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:52.944849 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:52.945957 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.946707 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.948330 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.948784 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.948880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:52.950227 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:52.950232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:52.950235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:52.950254 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:52.950288 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:52.950300 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:52.950396 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:52.950626 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 06:49:52.950756 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 06:49:52.952284 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:52.941439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38821 (local address 127.27.54.67:33127)
0807 06:49:52.942569 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:52.942615 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:52.944451 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:52.945093 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:52.945105 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:52.945199 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:52.945806 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:52.945823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.946879 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.946883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:52.948169 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:52.948182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.948971 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.948976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:52.949225 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:52.949917 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:49:52.949920 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:52.949958 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:49:52.950047 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:52.950395 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:52.950403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:52.950409 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:49:52.950554 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 06:49:52.950924 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":378,"threads_started":1}
I20260807 06:49:52.953794 9917 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38821
I20260807 06:49:52.954108 9917 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:52.955087 9917 heartbeater.cc:511] Master 127.27.54.126:38821 requested a full tablet report, sending...
I20260807 06:49:52.956761 9243 ts_manager.cc:194] Registered new tserver with Master: 76aa36f2b59047f8911a18ea17e645ac (127.27.54.67:35531)
I20260807 06:49:52.957576 9243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.67:33127
I20260807 06:49:52.969064 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 06:49:52.975841 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:52.974035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.27.54.126:38821)
0807 06:49:52.974243 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:52.974250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:52.974644 (+ 394us) server_negotiation.cc:407] Connection header received
0807 06:49:52.974877 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:52.974882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:52.974960 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:52.975068 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:52.975583 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36806: BlockingRecv error: recv got EOF from 127.0.0.1:36806 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 06:49:52.976603 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 9331
I20260807 06:49:52.984808 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:52.977654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.27.54.65:35205)
0807 06:49:52.978250 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:52.978257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:52.978286 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:49:52.978345 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:52.978351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:52.978561 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:52.978793 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:52.979178 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.980313 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.981634 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:52.982207 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:52.982309 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:52.983399 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:52.983407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:52.983420 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:52.984025 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:52.984072 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:52.984092 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:52.984193 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:52.984285 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 06:49:52.984398 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":288,"threads_started":1}
I20260807 06:49:52.986056 9513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:53.069259 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 9331
I20260807 06:49:53.094367 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 9528
I20260807 06:49:53.103076 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:53.095690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.27.54.66:38809)
0807 06:49:53.096336 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:53.096344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:53.096374 (+ 30us) server_negotiation.cc:407] Connection header received
0807 06:49:53.096444 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:53.096450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:53.096676 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:53.096829 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:53.097207 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:53.098248 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:53.099683 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:53.100302 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:53.100409 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:53.101560 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:53.101568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:53.101581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:53.102233 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:53.102284 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:53.102304 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:53.102418 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:53.102544 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 06:49:53.102719 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":339,"threads_started":1}
I20260807 06:49:53.104369 9709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:53.188685 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 9528
I20260807 06:49:53.214010 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 9721
I20260807 06:49:53.222445 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:53.215191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.27.54.67:35531)
0807 06:49:53.215872 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:53.215878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:53.215910 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:49:53.215975 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:53.215981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:53.216199 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:53.216342 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:53.216769 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:53.217852 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:53.219170 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:53.219745 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:53.219854 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:53.220920 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:53.220928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:53.220941 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:53.221561 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:53.221611 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:53.221632 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:53.221721 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:53.221906 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 06:49:53.222041 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":349,"threads_started":1}
I20260807 06:49:53.223807 9910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:53.309995 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 9721
I20260807 06:49:53.336503 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 9197
I20260807 06:49:53.337785 9293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:53.443563 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 9197
2026-08-07T06:49:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2716 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T06:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:53Z Disabled control of system clock
I20260807 06:49:53.544390 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38287
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 06:49:53.904721 9943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:53.905040 9943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:53.905128 9943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:53.912518 9943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:53.912655 9943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:53.912695 9943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:53.912757 9943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:53.912801 9943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:53.922195 9943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:38287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:53.923523 9943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:53.925333 9943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:53.937783 9956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:53.937863 9955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:53.937835 9958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:53.938558 9943 server_base.cc:1034] running on GCE node
I20260807 06:49:53.939380 9943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:53.940811 9943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:53.942058 9943 hybrid_clock.cc:648] HybridClock initialized: now 1786085393941994 us; error 49 us; skew 500 ppm
I20260807 06:49:53.942466 9943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:53.948069 9943 webserver.cc:460] Webserver started at http://127.27.54.126:34433/ using document root <none> and password file <none>
I20260807 06:49:53.948853 9943 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:53.948977 9943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:53.949342 9943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:53.952535 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "91000376233e4e4e984c52e6f4716ba9"
format_stamp: "Formatted at 2026-08-07 06:49:53 on dist-test-slave-3kk6"
server_key: "eee2b155df91c53214262abb794bd653"
server_key_iv: "d256749b5b254b174ee1ccc8ba2f7dc0"
server_key_version: "encryptionkey@0"
I20260807 06:49:53.953423 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "91000376233e4e4e984c52e6f4716ba9"
format_stamp: "Formatted at 2026-08-07 06:49:53 on dist-test-slave-3kk6"
server_key: "eee2b155df91c53214262abb794bd653"
server_key_iv: "d256749b5b254b174ee1ccc8ba2f7dc0"
server_key_version: "encryptionkey@0"
I20260807 06:49:53.959110 9943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:49:53.962585 9964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:53.963944 9943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:53.964133 9943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "91000376233e4e4e984c52e6f4716ba9"
format_stamp: "Formatted at 2026-08-07 06:49:53 on dist-test-slave-3kk6"
server_key: "eee2b155df91c53214262abb794bd653"
server_key_iv: "d256749b5b254b174ee1ccc8ba2f7dc0"
server_key_version: "encryptionkey@0"
I20260807 06:49:53.964357 9943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:53.970476 9943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:53.971294 9943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:53.971592 9943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:53.990825 9943 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:38287
I20260807 06:49:53.990863 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:38287 every 8 connection(s)
I20260807 06:49:53.992273 9943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:53.996287 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:53.999076 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 9943
I20260807 06:49:53.999296 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:54.000151 27865 external_mini_cluster.cc:1428] Setting key c4c89b7ff5bbef183e0c00915361fc79
I20260807 06:49:54.008884 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:54.002431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.27.54.126:38287)
0807 06:49:54.003551 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:54.003562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:54.003606 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:49:54.005261 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:54.005284 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:54.005695 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:54.006047 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:54.007269 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:54.007276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:54.007287 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:54.007302 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:54.007370 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:54.007388 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:54.007572 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:54.007676 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 06:49:54.007972 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":479,"threads_started":1}
I20260807 06:49:54.009653 10040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9: Bootstrap starting.
I20260807 06:49:54.013049 10040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:54.014248 10040 log.cc:824] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9: Log is configured to *not* fsync() on all Append() calls
I20260807 06:49:54.016969 10040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9: No bootstrap required, opened a new log
I20260807 06:49:54.024617 10040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91000376233e4e4e984c52e6f4716ba9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38287 } }
I20260807 06:49:54.024988 10040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:54.025087 10040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91000376233e4e4e984c52e6f4716ba9, State: Initialized, Role: FOLLOWER
I20260807 06:49:54.025640 10040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91000376233e4e4e984c52e6f4716ba9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38287 } }
I20260807 06:49:54.025872 10040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:54.026017 10040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:54.026199 10040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:54.028450 10040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91000376233e4e4e984c52e6f4716ba9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38287 } }
I20260807 06:49:54.029001 10040 leader_election.cc:302] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91000376233e4e4e984c52e6f4716ba9; no voters:
I20260807 06:49:54.029724 10040 leader_election.cc:288] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:54.029867 10046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:54.030616 10046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 91000376233e4e4e984c52e6f4716ba9, State: Running, Role: LEADER
I20260807 06:49:54.031193 10046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91000376233e4e4e984c52e6f4716ba9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38287 } }
I20260807 06:49:54.031888 10040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:54.034214 10048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91000376233e4e4e984c52e6f4716ba9. Latest consensus state: current_term: 1 leader_uuid: "91000376233e4e4e984c52e6f4716ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91000376233e4e4e984c52e6f4716ba9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38287 } } }
I20260807 06:49:54.034814 10048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:54.034262 10047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91000376233e4e4e984c52e6f4716ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91000376233e4e4e984c52e6f4716ba9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38287 } } }
I20260807 06:49:54.035384 10047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:54.037072 10066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:54.041311 10066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:54.049942 10066 catalog_manager.cc:1347] Generated new cluster ID: cbd93c194a714c3db0d534eaf29129cb
I20260807 06:49:54.050088 10066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:54.069391 10066 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:54.069886 10066 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:54.079105 10066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91000376233e4e4e984c52e6f4716ba9: Generated new TSK 0
I20260807 06:49:54.079959 10066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:49:54.092623 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38287
--builtin_ntp_servers=127.27.54.84:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 06:49:54.454114 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:54.454433 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:54.454628 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:54.462172 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:54.462297 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:54.462484 10078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:54.472149 10078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:54.473641 10078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:54.475584 10078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:54.489506 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:54.489399 10089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:54.489445 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:54.490854 10078 server_base.cc:1034] running on GCE node
I20260807 06:49:54.491472 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:54.492508 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:54.493729 10078 hybrid_clock.cc:648] HybridClock initialized: now 1786085394493678 us; error 50 us; skew 500 ppm
I20260807 06:49:54.494100 10078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:54.496634 10078 webserver.cc:460] Webserver started at http://127.27.54.65:41327/ using document root <none> and password file <none>
I20260807 06:49:54.497375 10078 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:54.497512 10078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:54.497877 10078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:54.500999 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "6cef003a23c84ff091af99f777d4f611"
format_stamp: "Formatted at 2026-08-07 06:49:54 on dist-test-slave-3kk6"
server_key: "065c03002efce9e79958e2073a9c330b"
server_key_iv: "0d9d2de8371f911587af855d55927038"
server_key_version: "encryptionkey@0"
I20260807 06:49:54.501876 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "6cef003a23c84ff091af99f777d4f611"
format_stamp: "Formatted at 2026-08-07 06:49:54 on dist-test-slave-3kk6"
server_key: "065c03002efce9e79958e2073a9c330b"
server_key_iv: "0d9d2de8371f911587af855d55927038"
server_key_version: "encryptionkey@0"
I20260807 06:49:54.507391 10078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260807 06:49:54.511363 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:54.512742 10078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260807 06:49:54.513021 10078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "6cef003a23c84ff091af99f777d4f611"
format_stamp: "Formatted at 2026-08-07 06:49:54 on dist-test-slave-3kk6"
server_key: "065c03002efce9e79958e2073a9c330b"
server_key_iv: "0d9d2de8371f911587af855d55927038"
server_key_version: "encryptionkey@0"
I20260807 06:49:54.513276 10078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:54.519232 10078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:54.520043 10078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:54.520344 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:54.521262 10078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:54.522729 10078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:54.522843 10078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:54.522974 10078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:54.523061 10078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:54.560729 10078 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:34481
I20260807 06:49:54.560772 10270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:34481 every 8 connection(s)
I20260807 06:49:54.562176 10078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:54.566331 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 10078
I20260807 06:49:54.566535 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:54.567512 27865 external_mini_cluster.cc:1428] Setting key 2c76292a04d6c3cdb372c82d10b61921
I20260807 06:49:54.570976 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38287
--builtin_ntp_servers=127.27.54.84:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 06:49:54.572438 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:54.564888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:57715 (local address 127.27.54.126:38287)
0807 06:49:54.565375 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:54.565380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:54.566962 (+ 1582us) server_negotiation.cc:407] Connection header received
0807 06:49:54.569266 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:54.569272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:54.569331 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:54.569471 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:54.571466 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:54.571475 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:54.571483 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:54.571508 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:54.571570 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:54.571589 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:54.571748 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:54.571960 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 06:49:54.572037 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":264,"threads_started":1}
I20260807 06:49:54.573366 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:54.565421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38287 (local address 127.27.54.65:57715)
0807 06:49:54.566759 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:54.566813 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:54.569008 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:54.569755 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:54.569765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:54.569911 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:54.571025 (+ 1114us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:49:54.571028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:54.571073 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:49:54.571245 (+ 172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:54.571733 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:54.571743 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:54.571749 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:49:54.572012 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 06:49:54.572243 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":357,"threads_started":1}
I20260807 06:49:54.575129 10271 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38287
I20260807 06:49:54.575471 10271 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:54.576133 10271 heartbeater.cc:511] Master 127.27.54.126:38287 requested a full tablet report, sending...
I20260807 06:49:54.578472 9988 ts_manager.cc:194] Registered new tserver with Master: 6cef003a23c84ff091af99f777d4f611 (127.27.54.65:34481)
W20260807 06:49:54.933885 10277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:54.934190 10277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:54.934284 10277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:54.941963 10277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:54.942098 10277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:54.942214 10277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:54.951882 10277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:54.953302 10277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:54.955256 10277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:54.969504 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:54.969515 10291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:54.969552 10287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:54.970608 10277 server_base.cc:1034] running on GCE node
I20260807 06:49:54.971448 10277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:54.972483 10277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:54.973752 10277 hybrid_clock.cc:648] HybridClock initialized: now 1786085394973688 us; error 48 us; skew 500 ppm
I20260807 06:49:54.974175 10277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:54.976934 10277 webserver.cc:460] Webserver started at http://127.27.54.66:37975/ using document root <none> and password file <none>
I20260807 06:49:54.977684 10277 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:54.977810 10277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:54.978156 10277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:54.981434 10277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "1bfeb0c1e1174a8ca426ec58e84da25c"
format_stamp: "Formatted at 2026-08-07 06:49:54 on dist-test-slave-3kk6"
server_key: "9a4ba4062022cc28f68959ff1e4b21c4"
server_key_iv: "07a14f55a55ee42c65a5a83c932a862a"
server_key_version: "encryptionkey@0"
I20260807 06:49:54.982331 10277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "1bfeb0c1e1174a8ca426ec58e84da25c"
format_stamp: "Formatted at 2026-08-07 06:49:54 on dist-test-slave-3kk6"
server_key: "9a4ba4062022cc28f68959ff1e4b21c4"
server_key_iv: "07a14f55a55ee42c65a5a83c932a862a"
server_key_version: "encryptionkey@0"
I20260807 06:49:54.988147 10277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 06:49:54.992275 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:54.993835 10277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 06:49:54.994026 10277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "1bfeb0c1e1174a8ca426ec58e84da25c"
format_stamp: "Formatted at 2026-08-07 06:49:54 on dist-test-slave-3kk6"
server_key: "9a4ba4062022cc28f68959ff1e4b21c4"
server_key_iv: "07a14f55a55ee42c65a5a83c932a862a"
server_key_version: "encryptionkey@0"
I20260807 06:49:54.994324 10277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:55.000806 10277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:55.001727 10277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:55.002034 10277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:55.003188 10277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:55.004869 10277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:55.004982 10277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:55.005113 10277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:55.005250 10277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 06:49:55.040609 10277 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:36181
I20260807 06:49:55.040664 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:36181 every 8 connection(s)
I20260807 06:49:55.041977 10277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:55.045361 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 10277
I20260807 06:49:55.045549 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:55.046173 27865 external_mini_cluster.cc:1428] Setting key b0618e2c0a08e602dca373d534610bee
I20260807 06:49:55.049453 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38287
--builtin_ntp_servers=127.27.54.84:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 06:49:55.051291 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.044731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:38359 (local address 127.27.54.126:38287)
0807 06:49:55.044908 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:55.044912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:55.046547 (+ 1635us) server_negotiation.cc:407] Connection header received
0807 06:49:55.048676 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:55.048680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:55.048760 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:55.048841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:55.050550 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:55.050555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:55.050558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:55.050575 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:55.050606 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:55.050617 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:55.050810 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:55.051025 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 06:49:55.051077 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:49:55.052695 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.045194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38287 (local address 127.27.54.66:38359)
0807 06:49:55.046347 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:55.046397 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:55.048360 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:55.049130 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:55.049144 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:55.049272 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:55.050176 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:49:55.050182 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:55.050237 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:49:55.050363 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:55.050790 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:55.050803 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:55.050814 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 06:49:55.051114 (+ 300us) client_negotiation.cc:240] Negotiation successful
0807 06:49:55.051389 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":338,"threads_started":1}
I20260807 06:49:55.054200 10460 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38287
I20260807 06:49:55.054512 10460 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:55.055209 10460 heartbeater.cc:511] Master 127.27.54.126:38287 requested a full tablet report, sending...
I20260807 06:49:55.056728 9988 ts_manager.cc:194] Registered new tserver with Master: 1bfeb0c1e1174a8ca426ec58e84da25c (127.27.54.66:36181)
W20260807 06:49:55.408182 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:55.408504 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:55.408684 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:55.416662 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:55.416848 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:55.417047 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:55.426693 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:55.428148 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:55.430059 10463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:55.444474 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:55.444517 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:55.444435 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:55.445441 10463 server_base.cc:1034] running on GCE node
I20260807 06:49:55.446121 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:55.447224 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:55.448465 10463 hybrid_clock.cc:648] HybridClock initialized: now 1786085395448415 us; error 62 us; skew 500 ppm
I20260807 06:49:55.448805 10463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:49:55.451298 10463 webserver.cc:460] Webserver started at http://127.27.54.67:40467/ using document root <none> and password file <none>
I20260807 06:49:55.452049 10463 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:55.452185 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:55.452529 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:55.455587 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "b04477da1f124f65b431f32657b8dcb1"
format_stamp: "Formatted at 2026-08-07 06:49:55 on dist-test-slave-3kk6"
server_key: "369ca99da2561d9086bdc54d02228f08"
server_key_iv: "b2e7b14e815ca3abad67e4474155103e"
server_key_version: "encryptionkey@0"
I20260807 06:49:55.456447 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "b04477da1f124f65b431f32657b8dcb1"
format_stamp: "Formatted at 2026-08-07 06:49:55 on dist-test-slave-3kk6"
server_key: "369ca99da2561d9086bdc54d02228f08"
server_key_iv: "b2e7b14e815ca3abad67e4474155103e"
server_key_version: "encryptionkey@0"
I20260807 06:49:55.461882 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:49:55.465476 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:55.466920 10463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:49:55.467106 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "b04477da1f124f65b431f32657b8dcb1"
format_stamp: "Formatted at 2026-08-07 06:49:55 on dist-test-slave-3kk6"
server_key: "369ca99da2561d9086bdc54d02228f08"
server_key_iv: "b2e7b14e815ca3abad67e4474155103e"
server_key_version: "encryptionkey@0"
I20260807 06:49:55.467348 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:55.473766 10463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:55.474617 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:55.474960 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:55.475927 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:55.477532 10463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:55.477658 10463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:55.477799 10463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:55.477890 10463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:55.511765 10463 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:36357
I20260807 06:49:55.511811 10641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:36357 every 8 connection(s)
I20260807 06:49:55.513168 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:49:55.521767 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.516103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:33297 (local address 127.27.54.126:38287)
0807 06:49:55.516292 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:55.516296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:55.517725 (+ 1429us) server_negotiation.cc:407] Connection header received
0807 06:49:55.519579 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:55.519585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:55.519653 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:55.519768 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:55.521009 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:55.521014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:55.521019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:55.521038 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:55.521077 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:55.521088 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:55.521210 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:55.521408 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 06:49:55.521481 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 06:49:55.522730 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.516584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38287 (local address 127.27.54.67:33297)
0807 06:49:55.517563 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:55.517605 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:55.519308 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:55.520005 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:55.520012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:55.520098 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:55.520703 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:49:55.520705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:55.520740 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:49:55.520821 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:55.521182 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:55.521194 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:55.521201 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:49:55.521490 (+ 289us) client_negotiation.cc:240] Negotiation successful
0807 06:49:55.521730 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":285,"threads_started":1}
I20260807 06:49:55.522722 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 10463
I20260807 06:49:55.522908 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:49:55.523375 27865 external_mini_cluster.cc:1428] Setting key 1cb683b7887c37baac97ef672808a522
I20260807 06:49:55.524210 10642 heartbeater.cc:347] Connected to a master server at 127.27.54.126:38287
I20260807 06:49:55.524564 10642 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:55.525381 10642 heartbeater.cc:511] Master 127.27.54.126:38287 requested a full tablet report, sending...
I20260807 06:49:55.526837 9988 ts_manager.cc:194] Registered new tserver with Master: b04477da1f124f65b431f32657b8dcb1 (127.27.54.67:36357)
I20260807 06:49:55.538942 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 06:49:55.545327 10274 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:49:55.543623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.27.54.126:38287)
0807 06:49:55.543828 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:55.543834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:55.544412 (+ 578us) server_negotiation.cc:407] Connection header received
0807 06:49:55.544565 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:55.544569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:55.544623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:55.544710 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:55.545032 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35668: BlockingRecv error: recv got EOF from 127.0.0.1:35668 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:49:55.545994 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 10078
I20260807 06:49:55.549602 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.547124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.27.54.65:34481)
0807 06:49:55.547599 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:55.547606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:55.547633 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:49:55.547693 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:55.547700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:55.547914 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:55.548040 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:55.548967 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:55.548974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:55.548987 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:55.549000 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:55.549047 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:55.549065 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:55.549183 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:55.549251 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 06:49:55.549354 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260807 06:49:55.550875 10263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:55.637364 10271 heartbeater.cc:503] Master 127.27.54.126:38287 was elected leader, sending a full tablet report...
I20260807 06:49:55.638540 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 10078
W20260807 06:49:55.663553 9972 connection.cc:535] server connection from 127.27.54.65:57715 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:49:55.663797 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 10277
I20260807 06:49:55.667927 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.664795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.27.54.66:36181)
0807 06:49:55.665352 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:55.665359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:55.665387 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:49:55.665456 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:55.665462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:55.665680 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:55.665906 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:55.667115 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:55.667121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:55.667133 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:55.667151 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:55.667190 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:55.667203 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:55.667366 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:55.667538 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 06:49:55.667650 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":246,"threads_started":1}
I20260807 06:49:55.669468 10455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:55.753443 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 10277
I20260807 06:49:55.778527 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 10463
I20260807 06:49:55.782933 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:55.779730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.27.54.67:36357)
0807 06:49:55.780409 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:55.780416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:55.780448 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:49:55.780513 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:55.780519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:55.780734 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:55.780875 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:55.781966 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:55.781973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:55.781983 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:49:55.782000 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:55.782039 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:49:55.782053 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:55.782189 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:55.782242 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 06:49:55.782373 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":355,"threads_started":1}
I20260807 06:49:55.784180 10634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:55.866961 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 10463
I20260807 06:49:55.891526 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 9943
I20260807 06:49:55.892809 10033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:49:55.998710 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 9943
2026-08-07T06:49:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2554 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:49:56 dist-test-slave-3kk6 krb5kdc[10665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:49:56 dist-test-slave-3kk6 krb5kdc[10665](info): set up 2 sockets
Aug 07 06:49:56 dist-test-slave-3kk6 krb5kdc[10665](info): commencing operation
krb5kdc: starting...
W20260807 06:49:57.350129 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:49:57 dist-test-slave-3kk6 krb5kdc[10665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:49:57.690326 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:36437
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:58.049649 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:58.049988 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:58.050081 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:58.057533 10684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:49:58.057680 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:58.057739 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:58.057792 10684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:49:58.057840 10684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:49:58.068353 10684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:36437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:58.069782 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:58.071136 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:58.084419 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:58.084503 10696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:58.084491 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:58.090969 10684 server_base.cc:1034] running on GCE node
I20260807 06:49:58.092007 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:58.093726 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:58.095011 10684 hybrid_clock.cc:648] HybridClock initialized: now 1786085398094929 us; error 60 us; skew 500 ppm
I20260807 06:49:58.095589 10684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:58 dist-test-slave-3kk6 krb5kdc[10665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085398, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:58.100739 10684 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:49:58.103726 10684 webserver.cc:460] Webserver started at http://127.27.54.126:43549/ using document root <none> and password file <none>
I20260807 06:49:58.104743 10684 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:58.104912 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:58.105222 10684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:58.108516 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "7b419935c23949548e97bed877b3a5d1"
format_stamp: "Formatted at 2026-08-07 06:49:58 on dist-test-slave-3kk6"
server_key: "c67cc2f4ec4046b4b1952b2dfc87c161"
server_key_iv: "9fb2143ea7740b7eedeab19853adadfa"
server_key_version: "encryptionkey@0"
I20260807 06:49:58.109304 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "7b419935c23949548e97bed877b3a5d1"
format_stamp: "Formatted at 2026-08-07 06:49:58 on dist-test-slave-3kk6"
server_key: "c67cc2f4ec4046b4b1952b2dfc87c161"
server_key_iv: "9fb2143ea7740b7eedeab19853adadfa"
server_key_version: "encryptionkey@0"
I20260807 06:49:58.115031 10684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:49:58.118597 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:58.120103 10684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:49:58.120311 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "7b419935c23949548e97bed877b3a5d1"
format_stamp: "Formatted at 2026-08-07 06:49:58 on dist-test-slave-3kk6"
server_key: "c67cc2f4ec4046b4b1952b2dfc87c161"
server_key_iv: "9fb2143ea7740b7eedeab19853adadfa"
server_key_version: "encryptionkey@0"
I20260807 06:49:58.120540 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:58.142724 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:58.147373 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:58.147703 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:58.167722 10684 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:36437
I20260807 06:49:58.167745 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:36437 every 8 connection(s)
I20260807 06:49:58.169202 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:49:58.173014 10786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:49:58.175454 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 10684
I20260807 06:49:58.175675 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:49:58.176501 27865 external_mini_cluster.cc:1428] Setting key ec56e8dec66a6c9e9bbf0107d6adeb4b
I20260807 06:49:58.186905 10786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1: Bootstrap starting.
I20260807 06:49:58.190186 10786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1: Neither blocks nor log segments found. Creating new log.
I20260807 06:49:58.191351 10786 log.cc:824] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:49:58 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:58.195047 10786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1: No bootstrap required, opened a new log
I20260807 06:49:58.202324 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:58.179005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.27.54.126:36437)
0807 06:49:58.181336 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:58.181349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:58.181391 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:49:58.183312 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:58.183346 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:58.183755 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:58.184188 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:58.185848 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:58.187348 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:58.188654 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:58.189348 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:58.189566 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:58.194525 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:58.194534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:58.194549 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:58.194584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:58.197920 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:58.198814 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:58.198822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:58.198828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:58.199007 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:58.199463 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:58.199467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:58.199469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:58.200021 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:58.200279 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:58.200776 (+ 497us) server_negotiation.cc:299] Negotiation successful
0807 06:49:58.201213 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2080,"thread_start_us":1837,"threads_started":1}
I20260807 06:49:58.204383 10786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b419935c23949548e97bed877b3a5d1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36437 } }
I20260807 06:49:58.204783 10786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:49:58.204919 10786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b419935c23949548e97bed877b3a5d1, State: Initialized, Role: FOLLOWER
I20260807 06:49:58.205559 10786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b419935c23949548e97bed877b3a5d1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36437 } }
I20260807 06:49:58.205823 10786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:49:58.205981 10786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:49:58.206168 10786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:49:58.208649 10786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b419935c23949548e97bed877b3a5d1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36437 } }
I20260807 06:49:58.209196 10786 leader_election.cc:302] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b419935c23949548e97bed877b3a5d1; no voters:
I20260807 06:49:58.209801 10786 leader_election.cc:288] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:49:58.209977 10794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:49:58.210793 10794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7b419935c23949548e97bed877b3a5d1, State: Running, Role: LEADER
I20260807 06:49:58.211448 10794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b419935c23949548e97bed877b3a5d1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36437 } }
I20260807 06:49:58.212316 10786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:49:58.214875 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b419935c23949548e97bed877b3a5d1. Latest consensus state: current_term: 1 leader_uuid: "7b419935c23949548e97bed877b3a5d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b419935c23949548e97bed877b3a5d1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36437 } } }
I20260807 06:49:58.215332 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:58.219738 10806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:49:58.219561 10795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b419935c23949548e97bed877b3a5d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b419935c23949548e97bed877b3a5d1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36437 } } }
I20260807 06:49:58.220011 10795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:49:58.225064 10806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:49:58.234117 10806 catalog_manager.cc:1347] Generated new cluster ID: cf7c565619af422c956bae2cd09b1f94
I20260807 06:49:58.234263 10806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:49:58.249372 10806 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:49:58.250761 10806 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:49:58.260267 10806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1: Generated new TSK 0
I20260807 06:49:58.261176 10806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:49:58.339051 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36437
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:58.698267 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:58.698623 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:58.698868 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:58.706265 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:58.706431 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:58.706633 10825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:49:58.717011 10825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:58.718436 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:58.720083 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:58.734197 10832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:58.734566 10836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:58.734627 10834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:58.735872 10825 server_base.cc:1034] running on GCE node
I20260807 06:49:58.736526 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:58.737597 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:58.738873 10825 hybrid_clock.cc:648] HybridClock initialized: now 1786085398738781 us; error 84 us; skew 500 ppm
I20260807 06:49:58.739204 10825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:58 dist-test-slave-3kk6 krb5kdc[10665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085398, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:58.742376 10825 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:49:58.743884 10825 webserver.cc:460] Webserver started at http://127.27.54.65:46579/ using document root <none> and password file <none>
I20260807 06:49:58.744606 10825 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:58.744755 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:58.745081 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:58.748318 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "5685bb0d4eb04fbaa004f147e3b39cb7"
format_stamp: "Formatted at 2026-08-07 06:49:58 on dist-test-slave-3kk6"
server_key: "73a5606cde6e7af85090a5c14db56e30"
server_key_iv: "7e670ef0173037ed9b527da070c26540"
server_key_version: "encryptionkey@0"
I20260807 06:49:58.749147 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "5685bb0d4eb04fbaa004f147e3b39cb7"
format_stamp: "Formatted at 2026-08-07 06:49:58 on dist-test-slave-3kk6"
server_key: "73a5606cde6e7af85090a5c14db56e30"
server_key_iv: "7e670ef0173037ed9b527da070c26540"
server_key_version: "encryptionkey@0"
I20260807 06:49:58.754583 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:49:58.758173 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:58.759603 10825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:58.759785 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "5685bb0d4eb04fbaa004f147e3b39cb7"
format_stamp: "Formatted at 2026-08-07 06:49:58 on dist-test-slave-3kk6"
server_key: "73a5606cde6e7af85090a5c14db56e30"
server_key_iv: "7e670ef0173037ed9b527da070c26540"
server_key_version: "encryptionkey@0"
I20260807 06:49:58.760082 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:58.773370 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:58.777541 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:58.777874 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:58.778934 10825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:58.780339 10825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:58.780447 10825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:58.780603 10825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:58.780678 10825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:58.815828 10825 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:38359
I20260807 06:49:58.815878 11006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:38359 every 8 connection(s)
I20260807 06:49:58.817207 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:49:58.821834 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 10825
I20260807 06:49:58.822067 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:49:58.822942 27865 external_mini_cluster.cc:1428] Setting key 598f4a46f44450d27aba8feb679f441a
Aug 07 06:49:58 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085398, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:58.840641 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:58.819948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:41297 (local address 127.27.54.126:36437)
0807 06:49:58.820463 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:58.820467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:58.821402 (+ 935us) server_negotiation.cc:407] Connection header received
0807 06:49:58.824209 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:58.824215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:58.824288 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:58.824436 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:58.826190 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:58.827444 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:58.829019 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:58.829714 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:58.829867 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:58.833498 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:58.833508 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:58.833516 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:58.833564 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:58.837193 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:58.837959 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:58.837967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:58.837973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:58.838147 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:58.838584 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:58.838592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:58.838597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:58.839019 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:58.839168 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:58.840186 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0807 06:49:58.840377 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260807 06:49:58.842253 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:58.820441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36437 (local address 127.27.54.65:41297)
0807 06:49:58.821165 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:58.821206 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:58.823891 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:58.824672 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:58.824681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:58.825333 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:58.826011 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:58.826031 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:58.827611 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:58.827615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:58.828860 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:58.828872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:58.829950 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:58.829955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:58.830165 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:58.830962 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:58.830991 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:58.833252 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:58.837356 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:58.837364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:58.837379 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:58.837833 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:58.838277 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:58.838280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:58.838283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:58.838473 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:58.839174 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:58.839184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:58.839742 (+ 558us) client_negotiation.cc:769] Sending connection context
0807 06:49:58.840101 (+ 359us) client_negotiation.cc:240] Negotiation successful
0807 06:49:58.840750 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":293,"threads_started":1}
I20260807 06:49:58.843998 11008 heartbeater.cc:347] Connected to a master server at 127.27.54.126:36437
I20260807 06:49:58.844337 11008 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:58.845131 11008 heartbeater.cc:511] Master 127.27.54.126:36437 requested a full tablet report, sending...
I20260807 06:49:58.847401 10735 ts_manager.cc:194] Registered new tserver with Master: 5685bb0d4eb04fbaa004f147e3b39cb7 (127.27.54.65:38359)
I20260807 06:49:58.848938 10735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:41297
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:49:58.896195 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36437
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:49:59.269814 11016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:59.270135 11016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:59.270315 11016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:59.277837 11016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:59.277976 11016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:59.278172 11016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:49:59.288426 11016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:59.290452 11016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:59.291954 11016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:59.307171 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:59.307214 11025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:59.307112 11022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:59.308126 11016 server_base.cc:1034] running on GCE node
I20260807 06:49:59.308838 11016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:59.309962 11016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:59.311165 11016 hybrid_clock.cc:648] HybridClock initialized: now 1786085399311091 us; error 60 us; skew 500 ppm
I20260807 06:49:59.311527 11016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:59 dist-test-slave-3kk6 krb5kdc[10665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085399, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:59.319998 11016 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:49:59.321724 11016 webserver.cc:460] Webserver started at http://127.27.54.66:35289/ using document root <none> and password file <none>
I20260807 06:49:59.322698 11016 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:59.322862 11016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:59.323243 11016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:59.326644 11016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "bff1bdefcbd34912b9184a91f54aad25"
format_stamp: "Formatted at 2026-08-07 06:49:59 on dist-test-slave-3kk6"
server_key: "fba737b84984392e473ecc655d1c54d8"
server_key_iv: "ef5582fbb6f5339b1eaec866282cb270"
server_key_version: "encryptionkey@0"
I20260807 06:49:59.327651 11016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "bff1bdefcbd34912b9184a91f54aad25"
format_stamp: "Formatted at 2026-08-07 06:49:59 on dist-test-slave-3kk6"
server_key: "fba737b84984392e473ecc655d1c54d8"
server_key_iv: "ef5582fbb6f5339b1eaec866282cb270"
server_key_version: "encryptionkey@0"
I20260807 06:49:59.333750 11016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 06:49:59.337731 11038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:59.339514 11016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:49:59.339733 11016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "bff1bdefcbd34912b9184a91f54aad25"
format_stamp: "Formatted at 2026-08-07 06:49:59 on dist-test-slave-3kk6"
server_key: "fba737b84984392e473ecc655d1c54d8"
server_key_iv: "ef5582fbb6f5339b1eaec866282cb270"
server_key_version: "encryptionkey@0"
I20260807 06:49:59.339962 11016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:59.376813 11016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:59.381683 11016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:59.382072 11016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:59.383175 11016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:59.384744 11016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:59.384824 11016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:59.385018 11016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:59.385077 11016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:59.422498 11016 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:37105
I20260807 06:49:59.422616 11211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:37105 every 8 connection(s)
I20260807 06:49:59.424083 11016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:49:59.432091 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 11016
I20260807 06:49:59.432281 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:49:59.432765 27865 external_mini_cluster.cc:1428] Setting key d18d1d9263ae13046d14e64f77367ef2
Aug 07 06:49:59 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085399, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:49:59.448897 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:59.427087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:53781 (local address 127.27.54.126:36437)
0807 06:49:59.427575 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0807 06:49:59.427579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:49:59.428632 (+ 1053us) server_negotiation.cc:407] Connection header received
0807 06:49:59.430538 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:49:59.430543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:49:59.430617 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:49:59.430785 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:49:59.432512 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:59.433386 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:59.435494 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:49:59.435948 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:49:59.436053 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:59.440783 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:49:59.440793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:49:59.440801 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:49:59.440850 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:49:59.444823 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:59.445798 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:59.445807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:59.445813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:59.446003 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:49:59.446545 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:49:59.446554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:49:59.446560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:49:59.447011 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:49:59.447183 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:49:59.448093 (+ 910us) server_negotiation.cc:299] Negotiation successful
0807 06:49:59.448396 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260807 06:49:59.450254 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:49:59.427754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36437 (local address 127.27.54.66:53781)
0807 06:49:59.428454 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0807 06:49:59.428494 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:49:59.430239 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:49:59.431037 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:49:59.431046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:49:59.431602 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:49:59.432263 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:59.432298 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:59.433604 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:59.433610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:59.435318 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:49:59.435333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:49:59.436205 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:49:59.436214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:49:59.436574 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:49:59.437426 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:49:59.437473 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:49:59.440478 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:49:59.445046 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:59.445058 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:59.445085 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:59.445649 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:59.446178 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:49:59.446182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:49:59.446185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:49:59.446413 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:49:59.447177 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:49:59.447187 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:49:59.447601 (+ 414us) client_negotiation.cc:769] Sending connection context
0807 06:49:59.447955 (+ 354us) client_negotiation.cc:240] Negotiation successful
0807 06:49:59.448659 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":282,"threads_started":1}
I20260807 06:49:59.452296 11213 heartbeater.cc:347] Connected to a master server at 127.27.54.126:36437
I20260807 06:49:59.452658 11213 heartbeater.cc:464] Registering TS with master...
I20260807 06:49:59.453486 11213 heartbeater.cc:511] Master 127.27.54.126:36437 requested a full tablet report, sending...
I20260807 06:49:59.455359 10735 ts_manager.cc:194] Registered new tserver with Master: bff1bdefcbd34912b9184a91f54aad25 (127.27.54.66:37105)
I20260807 06:49:59.456910 10735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:53781
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:49:59.510579 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36437
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:49:59.852157 11008 heartbeater.cc:503] Master 127.27.54.126:36437 was elected leader, sending a full tablet report...
W20260807 06:49:59.882813 11222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:49:59.883129 11222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:49:59.883325 11222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:49:59.890916 11222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:49:59.891052 11222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:49:59.891235 11222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:49:59.901532 11222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:49:59.903017 11222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:49:59.904477 11222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:49:59.918767 11230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:59.918630 11234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:49:59.919135 11231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:49:59.920559 11222 server_base.cc:1034] running on GCE node
I20260807 06:49:59.921301 11222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:49:59.922501 11222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:49:59.923744 11222 hybrid_clock.cc:648] HybridClock initialized: now 1786085399923684 us; error 53 us; skew 500 ppm
I20260807 06:49:59.924103 11222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:49:59 dist-test-slave-3kk6 krb5kdc[10665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085399, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:49:59.927563 11222 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:49:59.929208 11222 webserver.cc:460] Webserver started at http://127.27.54.67:42901/ using document root <none> and password file <none>
I20260807 06:49:59.929991 11222 fs_manager.cc:359] Metadata directory not provided
I20260807 06:49:59.930158 11222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:49:59.930605 11222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:49:59.933979 11222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "ce73cdce6f234dbebec52afe7e4ba9bd"
format_stamp: "Formatted at 2026-08-07 06:49:59 on dist-test-slave-3kk6"
server_key: "bc9ba218ea5baab5bd89d19bca53acbe"
server_key_iv: "4137a1e13fcf992362d95782567f0634"
server_key_version: "encryptionkey@0"
I20260807 06:49:59.934930 11222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "ce73cdce6f234dbebec52afe7e4ba9bd"
format_stamp: "Formatted at 2026-08-07 06:49:59 on dist-test-slave-3kk6"
server_key: "bc9ba218ea5baab5bd89d19bca53acbe"
server_key_iv: "4137a1e13fcf992362d95782567f0634"
server_key_version: "encryptionkey@0"
I20260807 06:49:59.940873 11222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 06:49:59.944974 11243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:59.946594 11222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:49:59.946852 11222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "ce73cdce6f234dbebec52afe7e4ba9bd"
format_stamp: "Formatted at 2026-08-07 06:49:59 on dist-test-slave-3kk6"
server_key: "bc9ba218ea5baab5bd89d19bca53acbe"
server_key_iv: "4137a1e13fcf992362d95782567f0634"
server_key_version: "encryptionkey@0"
I20260807 06:49:59.947088 11222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:49:59.975996 11222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:49:59.980544 11222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:49:59.980912 11222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:49:59.982096 11222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:49:59.983687 11222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:49:59.983805 11222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:49:59.983959 11222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:49:59.984037 11222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:00.021483 11222 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:45253
I20260807 06:50:00.021548 11419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:45253 every 8 connection(s)
I20260807 06:50:00.023183 11222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:50:00.027326 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 11222
I20260807 06:50:00.027562 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:50:00.028182 27865 external_mini_cluster.cc:1428] Setting key 96b18832c071809f97a3fbb1e0798694
Aug 07 06:50:00 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085399, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:50:00.046695 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.026507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:38919 (local address 127.27.54.126:36437)
0807 06:50:00.027152 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.027157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.028259 (+ 1102us) server_negotiation.cc:407] Connection header received
0807 06:50:00.031021 (+ 2762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.031025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.031096 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.031192 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:00.033067 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.033937 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.035748 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.036173 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.036301 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.040201 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:00.040207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:00.040211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:00.040252 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:00.043201 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.044269 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.044281 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.044284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.044464 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.045039 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.045044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.045046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.045350 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:00.045476 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.046168 (+ 692us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.046366 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":388,"threads_started":1}
I20260807 06:50:00.047760 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.027101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36437 (local address 127.27.54.67:38919)
0807 06:50:00.028061 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:00.028105 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:00.030645 (+ 2540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:00.031523 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:00.031537 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:00.032222 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:50:00.032888 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.032907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.034120 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.034124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.035569 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.035583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.036410 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.036416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.036635 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.037392 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:50:00.037415 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:50:00.039933 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:50:00.043447 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:00.043456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:00.043472 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:00.044112 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:00.044596 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:00.044601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:00.044624 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:00.044912 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:00.045464 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:50:00.045470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:50:00.045832 (+ 362us) client_negotiation.cc:769] Sending connection context
0807 06:50:00.046073 (+ 241us) client_negotiation.cc:240] Negotiation successful
0807 06:50:00.046513 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":411,"threads_started":1}
I20260807 06:50:00.049284 11420 heartbeater.cc:347] Connected to a master server at 127.27.54.126:36437
I20260807 06:50:00.049619 11420 heartbeater.cc:464] Registering TS with master...
I20260807 06:50:00.050412 11420 heartbeater.cc:511] Master 127.27.54.126:36437 requested a full tablet report, sending...
I20260807 06:50:00.052031 10735 ts_manager.cc:194] Registered new tserver with Master: ce73cdce6f234dbebec52afe7e4ba9bd (127.27.54.67:45253)
I20260807 06:50:00.052865 10735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:38919
I20260807 06:50:00.057188 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:50:00.076848 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.062204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.27.54.126:36437)
0807 06:50:00.062421 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.062426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.063128 (+ 702us) server_negotiation.cc:407] Connection header received
0807 06:50:00.063331 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.063336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.063419 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.063517 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:00.065330 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.066146 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.067663 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.068129 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.068234 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.071217 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:00.071222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:00.071225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:00.071265 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:00.074067 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.074882 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.074888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.074890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.075040 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.075427 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.075432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.075434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.075837 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:00.076037 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.076388 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.076576 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 06:50:00.081602 10735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:50:00.089247 10735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:50:00.119112 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.106178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:37105 (local address 127.0.0.1:56678)
0807 06:50:00.106962 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:00.107021 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:00.107227 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:00.107902 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:00.108109 (+ 207us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:00.108147 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:00.108658 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.108675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.111135 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.111143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.114398 (+ 3255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.114432 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.117668 (+ 3236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.117679 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.117997 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.118074 (+ 77us) client_negotiation.cc:769] Sending connection context
0807 06:50:00.118258 (+ 184us) client_negotiation.cc:240] Negotiation successful
0807 06:50:00.118532 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":343,"threads_started":1}
I20260807 06:50:00.120487 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.106339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.27.54.66:37105)
0807 06:50:00.106985 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.106993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.107167 (+ 174us) server_negotiation.cc:407] Connection header received
0807 06:50:00.107448 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.107454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.107652 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.107879 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:00.108924 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.110920 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.114727 (+ 3807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.117003 (+ 2276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.117197 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.117485 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.119869 (+ 2384us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.120135 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":344,"threads_started":1}
I20260807 06:50:00.126098 11105 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9122b6714a4f95a34a4c584139c3f1 (DEFAULT_TABLE table=test-table [id=da73fcec65794b3aaed3e51e4e09ff25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:50:00.128012 11105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9122b6714a4f95a34a4c584139c3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:00.135880 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.108608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:45253 (local address 127.0.0.1:34572)
0807 06:50:00.120584 (+ 11976us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:00.120619 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:00.120787 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:00.127127 (+ 6340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:00.127132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:00.127162 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:00.127633 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.127646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.129957 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.129962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.132408 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.132428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.134837 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.134846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.135048 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.135090 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 06:50:00.135169 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 06:50:00.135324 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11799,"mutex_wait_us":36,"thread_start_us":11549,"threads_started":1}
I20260807 06:50:00.136409 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.108911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.27.54.65:38359)
0807 06:50:00.124357 (+ 15446us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.124371 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.124418 (+ 47us) server_negotiation.cc:407] Connection header received
0807 06:50:00.124560 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.124573 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.125262 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.125573 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:00.126583 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.128830 (+ 2247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.132272 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.134417 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.134582 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.134877 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.136015 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.136161 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15181,"thread_start_us":15960,"threads_started":1}
I20260807 06:50:00.135913 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.108260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:38359 (local address 127.0.0.1:57260)
0807 06:50:00.119455 (+ 11195us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:00.119476 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:00.119628 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:00.125993 (+ 6365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:00.126001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:00.126046 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:00.126326 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.126342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.129084 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.129094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.131994 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.132020 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.134975 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.134984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.135179 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.135211 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:50:00.135305 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 06:50:00.135454 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11093,"mutex_wait_us":32,"thread_start_us":26200,"threads_started":1}
I20260807 06:50:00.137544 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.109186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.27.54.67:45253)
0807 06:50:00.126162 (+ 16976us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.126171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.126218 (+ 47us) server_negotiation.cc:407] Connection header received
0807 06:50:00.126344 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.126353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.126577 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.127847 (+ 1270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:00.128243 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.129783 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.132674 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.134174 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.136716 (+ 2542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.136961 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.137029 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.137197 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16765,"thread_start_us":17594,"threads_started":1}
I20260807 06:50:00.140858 10918 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9122b6714a4f95a34a4c584139c3f1 (DEFAULT_TABLE table=test-table [id=da73fcec65794b3aaed3e51e4e09ff25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:50:00.141319 11318 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9122b6714a4f95a34a4c584139c3f1 (DEFAULT_TABLE table=test-table [id=da73fcec65794b3aaed3e51e4e09ff25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:50:00.142588 10918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9122b6714a4f95a34a4c584139c3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:00.143167 11318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9122b6714a4f95a34a4c584139c3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:00.148429 11451 tablet_bootstrap.cc:492] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Bootstrap starting.
I20260807 06:50:00.152343 11451 tablet_bootstrap.cc:654] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:00.153793 11451 log.cc:824] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Log is configured to *not* fsync() on all Append() calls
I20260807 06:50:00.156800 11451 tablet_bootstrap.cc:492] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: No bootstrap required, opened a new log
I20260807 06:50:00.157130 11451 ts_tablet_manager.cc:1397] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 06:50:00.160193 11453 tablet_bootstrap.cc:492] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Bootstrap starting.
I20260807 06:50:00.164331 11453 tablet_bootstrap.cc:654] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:00.165747 11453 log.cc:824] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:50:00.168107 11451 raft_consensus.cc:348] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.169008 11451 raft_consensus.cc:374] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:00.169193 11451 raft_consensus.cc:729] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff1bdefcbd34912b9184a91f54aad25, State: Initialized, Role: FOLLOWER
I20260807 06:50:00.169538 11455 tablet_bootstrap.cc:492] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Bootstrap starting.
I20260807 06:50:00.170104 11451 consensus_queue.cc:260] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.171384 11213 heartbeater.cc:503] Master 127.27.54.126:36437 was elected leader, sending a full tablet report...
I20260807 06:50:00.171958 11451 ts_tablet_manager.cc:1428] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260807 06:50:00.173431 11455 tablet_bootstrap.cc:654] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:00.174741 11455 log.cc:824] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Log is configured to *not* fsync() on all Append() calls
W20260807 06:50:00.176386 11214 tablet.cc:2369] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:50:00.180881 11453 tablet_bootstrap.cc:492] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: No bootstrap required, opened a new log
I20260807 06:50:00.181245 11453 ts_tablet_manager.cc:1397] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20260807 06:50:00.184237 11455 tablet_bootstrap.cc:492] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: No bootstrap required, opened a new log
I20260807 06:50:00.184544 11455 ts_tablet_manager.cc:1397] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260807 06:50:00.191607 11453 raft_consensus.cc:348] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.192076 11453 raft_consensus.cc:374] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:00.192212 11453 raft_consensus.cc:729] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5685bb0d4eb04fbaa004f147e3b39cb7, State: Initialized, Role: FOLLOWER
I20260807 06:50:00.192905 11453 consensus_queue.cc:260] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.194892 11453 ts_tablet_manager.cc:1428] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260807 06:50:00.196684 11455 raft_consensus.cc:348] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.197183 11455 raft_consensus.cc:374] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:00.197315 11455 raft_consensus.cc:729] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce73cdce6f234dbebec52afe7e4ba9bd, State: Initialized, Role: FOLLOWER
I20260807 06:50:00.198007 11455 consensus_queue.cc:260] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.199404 11420 heartbeater.cc:503] Master 127.27.54.126:36437 was elected leader, sending a full tablet report...
I20260807 06:50:00.199936 11455 ts_tablet_manager.cc:1428] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260807 06:50:00.275398 11422 tablet.cc:2369] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:50:00.317727 11458 raft_consensus.cc:482] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:50:00.318046 11458 raft_consensus.cc:504] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
W20260807 06:50:00.320011 11009 tablet.cc:2369] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:50:00.321041 11458 leader_election.cc:288] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5685bb0d4eb04fbaa004f147e3b39cb7 (127.27.54.65:38359), ce73cdce6f234dbebec52afe7e4ba9bd (127.27.54.67:45253)
I20260807 06:50:00.329701 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.320706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:45541 (local address 127.27.54.65:38359)
0807 06:50:00.320974 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.320984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.322292 (+ 1308us) server_negotiation.cc:407] Connection header received
0807 06:50:00.322443 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.322448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.322544 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.322746 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:00.323456 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.324410 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.326858 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.328312 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.328517 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.328602 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.329205 (+ 603us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.329349 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 06:50:00.329643 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.321270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:58059 (local address 127.27.54.67:45253)
0807 06:50:00.321473 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.321479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.321908 (+ 429us) server_negotiation.cc:407] Connection header received
0807 06:50:00.322165 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.322172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.322269 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.322385 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:00.323172 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.324033 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.326846 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.328312 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.328517 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.328602 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.329193 (+ 591us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.329349 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 06:50:00.329957 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.320731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:38359 (local address 127.27.54.66:45541)
0807 06:50:00.322165 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:00.322183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:00.322323 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:00.322726 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:00.322735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:00.322766 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:00.323257 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.323272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.324575 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.324581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.326692 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.326707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.328624 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.328630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.328903 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.329031 (+ 128us) client_negotiation.cc:769] Sending connection context
0807 06:50:00.329139 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 06:50:00.329362 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"mutex_wait_us":120,"thread_start_us":1032,"threads_started":1}
I20260807 06:50:00.329977 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.320656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:45253 (local address 127.27.54.66:58059)
0807 06:50:00.321738 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:00.321769 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:00.322002 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:00.322421 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:00.322426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:00.322456 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:00.322981 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.322998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.324191 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.324198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.326589 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:00.326607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.328623 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.328630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:00.328906 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.329028 (+ 122us) client_negotiation.cc:769] Sending connection context
0807 06:50:00.329139 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:50:00.329355 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":139,"thread_start_us":678,"threads_started":1}
I20260807 06:50:00.331468 11349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9122b6714a4f95a34a4c584139c3f1" candidate_uuid: "bff1bdefcbd34912b9184a91f54aad25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" is_pre_election: true
I20260807 06:50:00.331616 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9122b6714a4f95a34a4c584139c3f1" candidate_uuid: "bff1bdefcbd34912b9184a91f54aad25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" is_pre_election: true
I20260807 06:50:00.332227 11349 raft_consensus.cc:2454] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff1bdefcbd34912b9184a91f54aad25 in term 0.
I20260807 06:50:00.332248 10941 raft_consensus.cc:2454] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bff1bdefcbd34912b9184a91f54aad25 in term 0.
I20260807 06:50:00.333103 11039 leader_election.cc:302] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5685bb0d4eb04fbaa004f147e3b39cb7, bff1bdefcbd34912b9184a91f54aad25, ce73cdce6f234dbebec52afe7e4ba9bd; no voters:
I20260807 06:50:00.333624 11458 raft_consensus.cc:2781] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:50:00.333812 11458 raft_consensus.cc:482] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:50:00.333937 11458 raft_consensus.cc:3037] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:00.336752 11458 raft_consensus.cc:504] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.337509 11458 leader_election.cc:288] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [CANDIDATE]: Term 1 election: Requested vote from peers 5685bb0d4eb04fbaa004f147e3b39cb7 (127.27.54.65:38359), ce73cdce6f234dbebec52afe7e4ba9bd (127.27.54.67:45253)
I20260807 06:50:00.338058 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9122b6714a4f95a34a4c584139c3f1" candidate_uuid: "bff1bdefcbd34912b9184a91f54aad25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7"
I20260807 06:50:00.338141 11349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9122b6714a4f95a34a4c584139c3f1" candidate_uuid: "bff1bdefcbd34912b9184a91f54aad25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd"
I20260807 06:50:00.338310 10941 raft_consensus.cc:3037] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:00.338439 11349 raft_consensus.cc:3037] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:00.340862 10941 raft_consensus.cc:2454] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff1bdefcbd34912b9184a91f54aad25 in term 1.
I20260807 06:50:00.340991 11349 raft_consensus.cc:2454] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bff1bdefcbd34912b9184a91f54aad25 in term 1.
I20260807 06:50:00.341442 11043 leader_election.cc:302] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5685bb0d4eb04fbaa004f147e3b39cb7, bff1bdefcbd34912b9184a91f54aad25; no voters:
I20260807 06:50:00.341813 11458 raft_consensus.cc:2781] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:50:00.342381 11458 raft_consensus.cc:686] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 1 LEADER]: Becoming Leader. State: Replica: bff1bdefcbd34912b9184a91f54aad25, State: Running, Role: LEADER
I20260807 06:50:00.342988 11458 consensus_queue.cc:237] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } }
I20260807 06:50:00.348029 10733 catalog_manager.cc:5495] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 reported cstate change: term changed from 0 to 1, leader changed from <none> to bff1bdefcbd34912b9184a91f54aad25 (127.27.54.66). New cstate: current_term: 1 leader_uuid: "bff1bdefcbd34912b9184a91f54aad25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff1bdefcbd34912b9184a91f54aad25" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 37105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 } health_report { overall_health: UNKNOWN } } }
I20260807 06:50:00.392982 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.387187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.27.54.66:37105)
0807 06:50:00.387414 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.387419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.387439 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:50:00.387556 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.387560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.387651 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.387805 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:50:00.388220 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.388987 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.390577 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.391093 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.391206 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.391785 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:50:00.392361 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:50:00.392486 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.392656 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.392741 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 06:50:00.402127 11349 raft_consensus.cc:1261] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 1 FOLLOWER]: Refusing update from remote peer bff1bdefcbd34912b9184a91f54aad25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:50:00.402118 10941 raft_consensus.cc:1261] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 1 FOLLOWER]: Refusing update from remote peer bff1bdefcbd34912b9184a91f54aad25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:50:00.403215 11466 consensus_queue.cc:1035] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce73cdce6f234dbebec52afe7e4ba9bd" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:50:00.403630 11458 consensus_queue.cc:1035] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5685bb0d4eb04fbaa004f147e3b39cb7" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:50:00.442598 10733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:50:00.443064 10733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:50:00.448978 11472 mvcc.cc:205] Tried to move back new op lower bound from 7315805800036425728 to 7315805799808741376. Current Snapshot: MvccSnapshot[applied={T|T < 7315805800036425728}]
I20260807 06:50:00.449100 10733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b419935c23949548e97bed877b3a5d1: Sending DeleteTablet for 3 replicas of tablet 8d9122b6714a4f95a34a4c584139c3f1
I20260807 06:50:00.452852 10918 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9122b6714a4f95a34a4c584139c3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:50:00 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:57260
I20260807 06:50:00.455055 11318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9122b6714a4f95a34a4c584139c3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:50:00 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:34572
I20260807 06:50:00.455350 11105 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9122b6714a4f95a34a4c584139c3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:50:00 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:56678
I20260807 06:50:00.465201 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 10825
I20260807 06:50:00.483847 11486 tablet_replica.cc:330] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: stopping tablet replica
I20260807 06:50:00.484236 11489 tablet_replica.cc:330] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: stopping tablet replica
I20260807 06:50:00.484408 11486 raft_consensus.cc:2229] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:50:00.485014 11486 raft_consensus.cc:2258] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:50:00.488639 11486 ts_tablet_manager.cc:1905] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:50:00.489073 11489 raft_consensus.cc:2229] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:50:00.490216 11488 tablet_replica.cc:330] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: stopping tablet replica
I20260807 06:50:00.490639 11488 raft_consensus.cc:2229] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:50:00.491102 11489 raft_consensus.cc:2258] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:50:00.491191 11488 raft_consensus.cc:2258] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:50:00.493468 11488 ts_tablet_manager.cc:1905] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:50:00.496810 11486 ts_tablet_manager.cc:1918] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:50:00.497082 11486 log.cc:1196] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/8d9122b6714a4f95a34a4c584139c3f1
I20260807 06:50:00.497694 11486 ts_tablet_manager.cc:1939] T 8d9122b6714a4f95a34a4c584139c3f1 P 5685bb0d4eb04fbaa004f147e3b39cb7: Deleting consensus metadata
I20260807 06:50:00.498721 11489 ts_tablet_manager.cc:1905] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:50:00.499823 10716 catalog_manager.cc:4841] TS 5685bb0d4eb04fbaa004f147e3b39cb7 (127.27.54.65:38359): tablet 8d9122b6714a4f95a34a4c584139c3f1 (table test-table [id=da73fcec65794b3aaed3e51e4e09ff25]) successfully deleted
I20260807 06:50:00.500862 11488 ts_tablet_manager.cc:1918] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:50:00.501089 11488 log.cc:1196] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/8d9122b6714a4f95a34a4c584139c3f1
I20260807 06:50:00.501681 11488 ts_tablet_manager.cc:1939] T 8d9122b6714a4f95a34a4c584139c3f1 P ce73cdce6f234dbebec52afe7e4ba9bd: Deleting consensus metadata
I20260807 06:50:00.503794 10711 catalog_manager.cc:4841] TS ce73cdce6f234dbebec52afe7e4ba9bd (127.27.54.67:45253): tablet 8d9122b6714a4f95a34a4c584139c3f1 (table test-table [id=da73fcec65794b3aaed3e51e4e09ff25]) successfully deleted
Aug 07 06:50:00 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:50:00.507581 11489 ts_tablet_manager.cc:1918] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:50:00.507825 11489 log.cc:1196] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/8d9122b6714a4f95a34a4c584139c3f1
I20260807 06:50:00.508564 11489 ts_tablet_manager.cc:1939] T 8d9122b6714a4f95a34a4c584139c3f1 P bff1bdefcbd34912b9184a91f54aad25: Deleting consensus metadata
I20260807 06:50:00.510891 10715 catalog_manager.cc:4841] TS bff1bdefcbd34912b9184a91f54aad25 (127.27.54.66:37105): tablet 8d9122b6714a4f95a34a4c584139c3f1 (table test-table [id=da73fcec65794b3aaed3e51e4e09ff25]) successfully deleted
I20260807 06:50:00.515796 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.467269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.27.54.65:38359)
0807 06:50:00.467543 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.467552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.483026 (+ 15474us) server_negotiation.cc:407] Connection header received
0807 06:50:00.485296 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.485306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.485424 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.485531 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:00.493742 (+ 8211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.495100 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.500358 (+ 5258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.501059 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.501204 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.508144 (+ 6940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:00.508173 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:00.508198 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:00.508269 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:00.512376 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.513490 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.513499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.513501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.513711 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.514172 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.514176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.514178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.514573 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:00.514964 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.515280 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.515470 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 06:50:00.516554 11002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:00.658445 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 10825
I20260807 06:50:00.692119 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 11016
Aug 07 06:50:00 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:50:00.708937 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.692998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.27.54.66:37105)
0807 06:50:00.693189 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.693195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.693274 (+ 79us) server_negotiation.cc:407] Connection header received
0807 06:50:00.693514 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.693524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.693604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.693727 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:00.695104 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.695961 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.697262 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.697718 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.697867 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.702382 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:00.702399 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:00.702415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:00.702463 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:00.705784 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.706591 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.706598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.706601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.706821 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.707262 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.707271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.707275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.707660 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:00.708053 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.708300 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.708543 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 06:50:00.709759 11206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:00.827288 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 11016
I20260807 06:50:00.854822 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 11222
Aug 07 06:50:00 dist-test-slave-3kk6 krb5kdc[10665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:50:00.872153 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:00.855632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.27.54.67:45253)
0807 06:50:00.856116 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:00.856120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:00.856138 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:50:00.856197 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:00.856202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:00.856270 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:00.856369 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:00.857846 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.858588 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.860135 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:00.860565 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:00.860680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:00.865724 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:00.865741 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:00.865758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:00.865805 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:00.868917 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.869739 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.869747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.869749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.869942 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:00.870402 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:00.870406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:00.870408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:00.870899 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:00.871143 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:00.871493 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 06:50:00.871701 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":274,"threads_started":1}
I20260807 06:50:00.872975 11413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:00.965714 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 11222
I20260807 06:50:00.991957 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 10684
I20260807 06:50:00.993207 10781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:01.107371 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 10684
2026-08-07T06:50:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5112 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:50:01 dist-test-slave-3kk6 krb5kdc[11502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:50:01 dist-test-slave-3kk6 krb5kdc[11502](info): set up 2 sockets
Aug 07 06:50:01 dist-test-slave-3kk6 krb5kdc[11502](info): commencing operation
krb5kdc: starting...
W20260807 06:50:02.460188 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:50:02 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:50:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/rangeradmin_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/rangeradmin_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/rangerlookup_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/rangerlookup_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
I20260807 06:50:02.963357 27865 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHOwvC2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/postgres -l logfile start
2026-08-07 06:50:05.785 UTC [11542] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 06:50:05.785 UTC [11542] LOG: listening on IPv4 address "127.27.54.84", port 33269
2026-08-07 06:50:05.790 UTC [11542] LOG: listening on Unix socket "/tmp/.s.PGSQL.33269"
2026-08-07 06:50:05.817 UTC [11546] LOG: database system was shut down at 2026-08-07 06:50:04 UTC
2026-08-07 06:50:05.821 UTC [11542] LOG: database system is ready to accept connections
I20260807 06:50:06.997885 27865 mini_postgres.cc:96] Postgres bound to 33269
2026-08-07 06:50:07.027 UTC [11559] FATAL: database "slave" does not exist
127.27.54.84:33269 - accepting connections
I20260807 06:50:07.028149 27865 mini_ranger.cc:161] Starting Ranger...
I20260807 06:50:07.076020 27865 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 06:50:07.629103 27865 mini_ranger.cc:87] Created ranger Postgres database
I20260807 06:50:07.629300 27865 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-admin
2026-08-07 06:50:08,000 [I] DB FLAVOR :POSTGRES
2026-08-07 06:50:08,001 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:50:08,001 [I] Checking connection..
2026-08-07 06:50:08,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 06:50:08,287 [I] Checking connection passed.
2026-08-07 06:50:08,287 [I] --------- Verifying version history table ---------
2026-08-07 06:50:08,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:50:08,551 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:50:08,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:50:08,827 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:50:08,827 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 06:50:08,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 06:50:09.101 UTC [11652] WARNING: there is no transaction in progress
2026-08-07 06:50:09,111 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 06:50:09,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:50:09,378 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 06:50:09,379 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:50:09,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 06:50:09,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 06:50:09,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3kk6.c.gcp-upstream.internal','N') ;"
2026-08-07 06:50:10,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:50:10,444 [I] Table x_portal_user does not exist in database ranger
2026-08-07 06:50:10,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:50:10,716 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 06:50:10,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:50:10,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:50:11,257 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 06:50:11,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 06:50:12.349 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.368 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.387 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.404 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.422 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.445 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.512 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.520 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.536 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.548 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.560 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.575 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.586 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.594 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.602 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:12.611 UTC [11813] WARNING: there is no transaction in progress
2026-08-07 06:50:13,307 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 06:50:13,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 06:50:13,578 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 06:50:13,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:50:13,842 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 06:50:13,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:50:14,087 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 06:50:14,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:50:14,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:50:14,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3kk6.c.gcp-upstream.internal';"
2026-08-07 06:50:14,863 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 06:50:14,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 06:50:15,137 [I] DB_PATCHES have already been applied
I20260807 06:50:15.144825 27865 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHOwvC2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 06:50:15.145009 27865 mini_ranger.cc:193] Using host: 127.27.54.84
I20260807 06:50:15.170953 27865 mini_ranger.cc:227] Ranger admin URL: http://127.27.54.84:53683
Aug 07, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 6:50:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 6:50:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 06:50:16.565783 27865 mini_ranger.cc:160] Time spent starting Ranger: real 9.538s user 0.000s sys 0.085s
Aug 07, 2026 6:50:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 6:50:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 6:50:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 6:50:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 6:50:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 06:50:38.913347 27865 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/rangerkms_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/rangerkms_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.54.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.54.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 06:50:39 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085439, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 06:50:39.182610 27865 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 06:50:39.215688 27865 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 06:50:40.139293 27865 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 06:50:40.139457 27865 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms
I20260807 06:50:40.139642 27865 mini_ranger_kms.cc:227] Using postgres at 127.27.54.84:33269
2026-08-07 06:50:40,707 [I] DB FLAVOR :POSTGRES
2026-08-07 06:50:40,707 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:50:40,707 [I] Checking connection
2026-08-07 06:50:40,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 06:50:40,978 [I] connection success
2026-08-07 06:50:40,978 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 06:50:40,979 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 06:50:40,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 06:50:41,247 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 06:50:41,247 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:50:41,247 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 06:50:41,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:33269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 06:50:41,549 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 06:50:42.133796 27865 mini_ranger_kms.cc:310] Created kms service
I20260807 06:50:42.373656 27865 mini_ranger_kms.cc:326] Created kudu user
I20260807 06:50:42.427675 27865 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 06:50:42 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085439, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:42.894243 27865 mini_ranger_kms.cc:384] Added ranger policy
I20260807 06:50:42.894774 27865 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 06:50:42.894869 27865 mini_ranger_kms.cc:240] Using host: 127.27.54.84
I20260807 06:50:42.931888 27865 mini_ranger_kms.cc:276] Ranger KMS PID: 12079
I20260807 06:50:42.932125 27865 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.54.84:55037
Aug 07, 2026 6:50:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:50:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:50:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 6:50:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 6:50:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 6:50:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 06:50:45 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786085445, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.54.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 06:50:45 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085445, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.54.84@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:45.852916 27865 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.670s user 0.136s sys 0.492s
I20260807 06:50:45.853320 27865 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 06:50:45.853544 27865 mini_ranger_kms.cc:401] 127.27.54.84:55037/kms/v1/keys
Aug 07 06:50:46 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085445, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.54.84@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
Aug 07 06:50:48 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:50:48.811982 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:46243
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:46243
--ranger_config_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:55037/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:50:49.221220 12149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:50:49.221659 12149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:50:49.221869 12149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:50:49.230334 12149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:50:49.230566 12149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:50:49.230720 12149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:50:49.230844 12149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:50:49.230931 12149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:50:49.242535 12149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.54.84:55037/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:46243
--ranger_config_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:50:49.244217 12149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:50:49.245795 12149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:50:49.259846 12155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:49.259852 12156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:49.259790 12160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:50:49.260651 12149 server_base.cc:1034] running on GCE node
I20260807 06:50:49.261687 12149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:50:49.263536 12149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:50:49.264815 12149 hybrid_clock.cc:648] HybridClock initialized: now 1786085449264741 us; error 69 us; skew 500 ppm
I20260807 06:50:49.265205 12149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:50:49 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085449, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:50:49.269012 12149 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:50:49.270807 12149 webserver.cc:460] Webserver started at http://127.27.54.126:35869/ using document root <none> and password file <none>
I20260807 06:50:49.271533 12149 fs_manager.cc:359] Metadata directory not provided
I20260807 06:50:49.271718 12149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:50:49.272068 12149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:50:49 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085449, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:49.330511 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "af98b5287e254174825201cd8652d3e4"
format_stamp: "Formatted at 2026-08-07 06:50:49 on dist-test-slave-3kk6"
server_key: "dc5269c0b84fdc54895f6d70575c6878"
server_key_iv: "b1c23732b8d5e10bf85d29d160a74e88"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:49.331759 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "af98b5287e254174825201cd8652d3e4"
format_stamp: "Formatted at 2026-08-07 06:50:49 on dist-test-slave-3kk6"
server_key: "dc5269c0b84fdc54895f6d70575c6878"
server_key_iv: "b1c23732b8d5e10bf85d29d160a74e88"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:49.338444 12149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 06:50:49.359542 12170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:49.361364 12149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:50:49.361560 12149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "af98b5287e254174825201cd8652d3e4"
format_stamp: "Formatted at 2026-08-07 06:50:49 on dist-test-slave-3kk6"
server_key: "dc5269c0b84fdc54895f6d70575c6878"
server_key_iv: "b1c23732b8d5e10bf85d29d160a74e88"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:49.361825 12149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:50:49.385015 12149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:50:49.389751 12149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:50:49.390152 12149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:50:49.414897 12149 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:46243
I20260807 06:50:49.414940 12243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:46243 every 8 connection(s)
I20260807 06:50:49.416477 12149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:50:49.420851 12244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:49.426358 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 12149
I20260807 06:50:49.426611 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:50:49.427584 27865 external_mini_cluster.cc:1428] Setting key f67843ea9265f67ea375475a7d764252
I20260807 06:50:49.434283 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4: Bootstrap starting.
I20260807 06:50:49.438715 12244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:49.440294 12244 log.cc:824] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:50:49 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:50:49.443915 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4: No bootstrap required, opened a new log
I20260807 06:50:49.453445 12244 raft_consensus.cc:348] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98b5287e254174825201cd8652d3e4" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46243 } }
I20260807 06:50:49.453922 12244 raft_consensus.cc:374] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:49.454118 12244 raft_consensus.cc:729] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af98b5287e254174825201cd8652d3e4, State: Initialized, Role: FOLLOWER
I20260807 06:50:49.454155 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:49.430113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.27.54.126:46243)
0807 06:50:49.431261 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:49.431273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:49.431323 (+ 50us) server_negotiation.cc:407] Connection header received
0807 06:50:49.433207 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:49.433237 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:49.433670 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:49.434053 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:49.436032 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:49.437578 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:49.439602 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:49.440431 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:49.440667 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:49.445044 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:49.445053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:49.445069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:49.445119 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:49.448920 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:49.449821 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:49.449832 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:49.449839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:49.450010 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:49.450471 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:49.450475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:49.450477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:49.451198 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:49.451570 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:49.452113 (+ 543us) server_negotiation.cc:299] Negotiation successful
0807 06:50:49.452649 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":602,"threads_started":1}
I20260807 06:50:49.455024 12244 consensus_queue.cc:260] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98b5287e254174825201cd8652d3e4" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46243 } }
I20260807 06:50:49.455380 12244 raft_consensus.cc:388] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:50:49.455550 12244 raft_consensus.cc:482] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:50:49.455783 12244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:49.459015 12244 raft_consensus.cc:504] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98b5287e254174825201cd8652d3e4" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46243 } }
I20260807 06:50:49.459610 12244 leader_election.cc:302] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af98b5287e254174825201cd8652d3e4; no voters:
I20260807 06:50:49.460383 12244 leader_election.cc:288] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:50:49.460544 12251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:50:49.462028 12251 raft_consensus.cc:686] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [term 1 LEADER]: Becoming Leader. State: Replica: af98b5287e254174825201cd8652d3e4, State: Running, Role: LEADER
I20260807 06:50:49.463032 12244 sys_catalog.cc:564] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:50:49.462924 12251 consensus_queue.cc:237] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98b5287e254174825201cd8652d3e4" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46243 } }
I20260807 06:50:49.466243 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af98b5287e254174825201cd8652d3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98b5287e254174825201cd8652d3e4" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46243 } } }
I20260807 06:50:49.466279 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af98b5287e254174825201cd8652d3e4. Latest consensus state: current_term: 1 leader_uuid: "af98b5287e254174825201cd8652d3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af98b5287e254174825201cd8652d3e4" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46243 } } }
I20260807 06:50:49.467013 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:50:49.467013 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4 [sys.catalog]: This master's current role is: LEADER
I20260807 06:50:49.469877 12259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:50:49.477385 12259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:50:49.480573 12244 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 06:50:49.502493 12259 catalog_manager.cc:1347] Generated new cluster ID: 4c20f552ac504ef49f37314f6eec6a8c
I20260807 06:50:49.502708 12259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:50:49.540880 12259 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:50:49.543318 12259 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:50:49.559224 12259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4: Generated new TSK 0
I20260807 06:50:49.560627 12259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:50:50.455138 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:55037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46243
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:50:50 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786085450, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 06:50:50.920715 12306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:50:50.921167 12306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:50:50.921406 12306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:50:50.933583 12306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:50:50.933841 12306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:50:50.934072 12306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:50:50.950775 12306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.54.84:55037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:50:50.952764 12306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:50:50.954703 12306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:50:50.972404 12319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:50.972380 12320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:50.973361 12322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:50:50.974445 12306 server_base.cc:1034] running on GCE node
I20260807 06:50:50.975188 12306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:50:50.976351 12306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:50:50.977667 12306 hybrid_clock.cc:648] HybridClock initialized: now 1786085450977589 us; error 74 us; skew 500 ppm
I20260807 06:50:50.978021 12306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:50:50 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085450, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:50:50.982105 12306 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:50:50.994292 12306 webserver.cc:460] Webserver started at http://127.27.54.65:44079/ using document root <none> and password file <none>
I20260807 06:50:50.995416 12306 fs_manager.cc:359] Metadata directory not provided
I20260807 06:50:50.995637 12306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:50:50.996071 12306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:50:50 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085450, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:51.019747 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "78d7062ea26147b4a7467a599d734837"
format_stamp: "Formatted at 2026-08-07 06:50:51 on dist-test-slave-3kk6"
server_key: "649ef77fd7d9d59b88badf193e9f1ebb"
server_key_iv: "2bdd11d775c3a224f8fa16c68253c77d"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:51.021111 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "78d7062ea26147b4a7467a599d734837"
format_stamp: "Formatted at 2026-08-07 06:50:51 on dist-test-slave-3kk6"
server_key: "649ef77fd7d9d59b88badf193e9f1ebb"
server_key_iv: "2bdd11d775c3a224f8fa16c68253c77d"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:51.028347 12306 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 06:50:51.047333 12332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:51.049170 12306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:50:51.049410 12306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "78d7062ea26147b4a7467a599d734837"
format_stamp: "Formatted at 2026-08-07 06:50:51 on dist-test-slave-3kk6"
server_key: "649ef77fd7d9d59b88badf193e9f1ebb"
server_key_iv: "2bdd11d775c3a224f8fa16c68253c77d"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:51.049680 12306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:50:51.088512 12306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:50:51.094229 12306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:50:51.094772 12306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:50:51.098608 12306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:50:51.100589 12306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:50:51.100715 12306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:51.100858 12306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:50:51.100922 12306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:51.201696 12306 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:33681
I20260807 06:50:51.201697 12485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:33681 every 8 connection(s)
I20260807 06:50:51.203310 12306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:50:51.212841 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 12306
I20260807 06:50:51.213112 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:50:51.214020 27865 external_mini_cluster.cc:1428] Setting key 4eb4dd55fdf3ffb1a290f53314b53491
Aug 07 06:50:51 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085450, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:50:51.260584 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:51.208170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46243 (local address 127.27.54.65:55471)
0807 06:50:51.220311 (+ 12141us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:51.220392 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:51.224369 (+ 3977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:51.225684 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:51.225699 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:51.226497 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:50:51.227548 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:51.227581 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:51.229287 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:51.229293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:51.231558 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:51.231577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:51.233108 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:51.233117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:51.233458 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:51.237848 (+ 4390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:50:51.237884 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:50:51.240878 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:50:51.254574 (+ 13696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:51.254589 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:51.254615 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:51.255511 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:51.256318 (+ 807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:51.256324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:51.256328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:51.256673 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:51.257483 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:50:51.257493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:50:51.258033 (+ 540us) client_negotiation.cc:769] Sending connection context
0807 06:50:51.258355 (+ 322us) client_negotiation.cc:240] Negotiation successful
0807 06:50:51.259030 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11751,"thread_start_us":14351,"threads_started":1}
I20260807 06:50:51.262404 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:51.207500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:55471 (local address 127.27.54.126:46243)
0807 06:50:51.219583 (+ 12083us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:51.219590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:51.223743 (+ 4153us) server_negotiation.cc:407] Connection header received
0807 06:50:51.224786 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:51.224793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:51.224935 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:51.225039 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:51.227806 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:51.229042 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:51.231789 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:51.232559 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:51.232674 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:51.241233 (+ 8559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:51.241241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:51.241246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:51.241309 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:51.254293 (+ 12984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:51.255782 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:51.255789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:51.255796 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:51.256156 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:51.256858 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:51.256864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:51.256870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:51.257312 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:51.261492 (+ 4180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:51.261638 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 06:50:51.261959 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11921,"thread_start_us":19161,"threads_started":1}
I20260807 06:50:51.264183 12486 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46243
I20260807 06:50:51.264544 12486 heartbeater.cc:464] Registering TS with master...
I20260807 06:50:51.265444 12486 heartbeater.cc:511] Master 127.27.54.126:46243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
I20260807 06:50:51.268515 12195 ts_manager.cc:194] Registered new tserver with Master: 78d7062ea26147b4a7467a599d734837 (127.27.54.65:33681)
Aug 07 06:50:51 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085450, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:51.271229 12195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:55471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
W20260807 06:50:51.332816 27865 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.54.66: real 0.116s user 0.002s sys 0.025s
I20260807 06:50:51.333947 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:55037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46243
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:50:51 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085450, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
W20260807 06:50:51.844486 12492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:50:51.844875 12492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:50:51.845057 12492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:50:51.852883 12492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:50:51.853142 12492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:50:51.853339 12492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:50:51.864779 12492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.54.84:55037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:50:51.866364 12492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:50:51.868345 12492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:50:51.897672 12503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:51.899519 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:51.899842 12505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:50:51.901616 12492 server_base.cc:1034] running on GCE node
I20260807 06:50:51.902530 12492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:50:51.903963 12492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:50:51.905262 12492 hybrid_clock.cc:648] HybridClock initialized: now 1786085451905215 us; error 45 us; skew 500 ppm
I20260807 06:50:51.905684 12492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:50:51 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:50:51.910800 12492 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:50:51.916224 12492 webserver.cc:460] Webserver started at http://127.27.54.66:39207/ using document root <none> and password file <none>
I20260807 06:50:51.917092 12492 fs_manager.cc:359] Metadata directory not provided
I20260807 06:50:51.917263 12492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:50:51.917641 12492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:50:51 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:51.936355 12492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "1c7fb90168b74d1b9fc03d612633a9ae"
format_stamp: "Formatted at 2026-08-07 06:50:51 on dist-test-slave-3kk6"
server_key: "351757ad790aa58952a802411e3e9ed0"
server_key_iv: "0add835aa0ec0abda7340c43f4ffa970"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:51.937346 12492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "1c7fb90168b74d1b9fc03d612633a9ae"
format_stamp: "Formatted at 2026-08-07 06:50:51 on dist-test-slave-3kk6"
server_key: "351757ad790aa58952a802411e3e9ed0"
server_key_iv: "0add835aa0ec0abda7340c43f4ffa970"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:51.944133 12492 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 06:50:51.966398 12523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:51.968585 12492 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.007s sys 0.000s
I20260807 06:50:51.968801 12492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "1c7fb90168b74d1b9fc03d612633a9ae"
format_stamp: "Formatted at 2026-08-07 06:50:51 on dist-test-slave-3kk6"
server_key: "351757ad790aa58952a802411e3e9ed0"
server_key_iv: "0add835aa0ec0abda7340c43f4ffa970"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:51.969075 12492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:50:51.995921 12492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:50:52.001210 12492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:50:52.001590 12492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:50:52.002965 12492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:50:52.004848 12492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:50:52.004947 12492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:52.005065 12492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:50:52.005129 12492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:52.050153 12492 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:45423
I20260807 06:50:52.050180 12686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:45423 every 8 connection(s)
I20260807 06:50:52.052374 12492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:50:52.053691 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 12492
I20260807 06:50:52.053939 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:50:52.054478 27865 external_mini_cluster.cc:1428] Setting key 1f3d7d8753208fa37882286b3414b4fa
Aug 07 06:50:52 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:50:52.085291 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.057406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:35703 (local address 127.27.54.126:46243)
0807 06:50:52.063129 (+ 5723us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:52.063134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:52.063155 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:50:52.063303 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:52.063307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:52.063390 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:52.063491 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:52.065740 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.066978 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.069485 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.070133 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.070339 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.075990 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:52.076000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:52.076009 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:52.076072 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:52.080824 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:52.081913 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:52.081923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:52.081930 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:52.082138 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:52.082750 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:52.082759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:52.082765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:52.083221 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:52.083403 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:52.084378 (+ 975us) server_negotiation.cc:299] Negotiation successful
0807 06:50:52.084674 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5614,"thread_start_us":5484,"threads_started":1}
I20260807 06:50:52.086851 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.058361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46243 (local address 127.27.54.66:35703)
0807 06:50:52.060152 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:52.060234 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:52.062877 (+ 2643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:52.063831 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:52.063845 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:52.064541 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:50:52.065484 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.065512 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.067219 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.067224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.069278 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.069295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.070775 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.070785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.071122 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.072113 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:50:52.072171 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:50:52.075696 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:50:52.081029 (+ 5333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:52.081040 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:52.081062 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:52.081740 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:52.082306 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:52.082311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:52.082315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:52.082581 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:52.083380 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:50:52.083388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:50:52.083857 (+ 469us) client_negotiation.cc:769] Sending connection context
0807 06:50:52.084193 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 06:50:52.084814 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":614,"threads_started":1}
I20260807 06:50:52.088847 12687 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46243
I20260807 06:50:52.089249 12687 heartbeater.cc:464] Registering TS with master...
I20260807 06:50:52.090353 12687 heartbeater.cc:511] Master 127.27.54.126:46243 requested a full tablet report, sending...
I20260807 06:50:52.092549 12195 ts_manager.cc:194] Registered new tserver with Master: 1c7fb90168b74d1b9fc03d612633a9ae (127.27.54.66:45423)
I20260807 06:50:52.093797 12195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:35703
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:50:52.152765 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:55037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46243
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:50:52.280797 12486 heartbeater.cc:503] Master 127.27.54.126:46243 was elected leader, sending a full tablet report...
W20260807 06:50:52.597402 12693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:50:52.597728 12693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:50:52.597854 12693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:50:52.605842 12693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:50:52.606001 12693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:50:52.606132 12693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:50:52.617185 12693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.54.84:55037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:50:52.618620 12693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:50:52.620165 12693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:50:52.635830 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:52.635969 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:50:52.635851 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:50:52.636909 12693 server_base.cc:1034] running on GCE node
I20260807 06:50:52.637614 12693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:50:52.638773 12693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:50:52.640015 12693 hybrid_clock.cc:648] HybridClock initialized: now 1786085452639967 us; error 49 us; skew 500 ppm
I20260807 06:50:52.640368 12693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:50:52 dist-test-slave-3kk6 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:50:52.646818 12693 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:50:52.648562 12693 webserver.cc:460] Webserver started at http://127.27.54.67:34363/ using document root <none> and password file <none>
I20260807 06:50:52.649458 12693 fs_manager.cc:359] Metadata directory not provided
I20260807 06:50:52.649586 12693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:50:52.649916 12693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:50:52 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:50:52.669654 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "ec04c9350d784c77a23e5177538baffb"
format_stamp: "Formatted at 2026-08-07 06:50:52 on dist-test-slave-3kk6"
server_key: "69b42b4196781ae5f7aac1b9cc2685ef"
server_key_iv: "df138637686103304ea22b9715b0f705"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:52.670899 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "ec04c9350d784c77a23e5177538baffb"
format_stamp: "Formatted at 2026-08-07 06:50:52 on dist-test-slave-3kk6"
server_key: "69b42b4196781ae5f7aac1b9cc2685ef"
server_key_iv: "df138637686103304ea22b9715b0f705"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:52.677450 12693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 06:50:52.696157 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:52.697710 12693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 06:50:52.697954 12693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "ec04c9350d784c77a23e5177538baffb"
format_stamp: "Formatted at 2026-08-07 06:50:52 on dist-test-slave-3kk6"
server_key: "69b42b4196781ae5f7aac1b9cc2685ef"
server_key_iv: "df138637686103304ea22b9715b0f705"
server_key_version: "kuduclusterkey@0"
I20260807 06:50:52.698230 12693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:50:52.733994 12693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:50:52.738773 12693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:50:52.739140 12693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:50:52.740397 12693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:50:52.742086 12693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:50:52.742175 12693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:52.742359 12693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:50:52.742442 12693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:50:52.787655 12693 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:33563
I20260807 06:50:52.787853 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:33563 every 8 connection(s)
I20260807 06:50:52.789145 12693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:50:52.792867 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 12693
I20260807 06:50:52.793061 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:50:52.793723 27865 external_mini_cluster.cc:1428] Setting key 439e016bbc5230cfdd80eb93e60cafc5
Aug 07 06:50:52 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:50:52.812837 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.791983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:60647 (local address 127.27.54.126:46243)
0807 06:50:52.792620 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:52.792627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:52.794070 (+ 1443us) server_negotiation.cc:407] Connection header received
0807 06:50:52.796771 (+ 2701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:52.796775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:52.796847 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:52.796948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:52.798927 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.799729 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.801310 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.801799 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.801900 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.805879 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:52.805885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:52.805888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:52.805929 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:52.809226 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:52.810321 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:52.810325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:52.810328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:52.810542 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:52.811096 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:52.811099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:52.811101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:52.811418 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:52.811551 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:52.812335 (+ 784us) server_negotiation.cc:299] Negotiation successful
0807 06:50:52.812549 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":268,"threads_started":1}
I20260807 06:50:52.813871 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.792731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46243 (local address 127.27.54.67:60647)
0807 06:50:52.793837 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:52.793906 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:52.796427 (+ 2521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:52.797378 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:52.797405 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:52.798027 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:50:52.798761 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.798784 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.799888 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.799892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.801169 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.801182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.802050 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.802056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.802333 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.803159 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:50:52.803181 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:50:52.805636 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:50:52.809426 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:52.809435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:52.809451 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:52.810159 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:52.810720 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:50:52.810725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:50:52.810728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:50:52.810977 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:50:52.811542 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:50:52.811549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:50:52.811969 (+ 420us) client_negotiation.cc:769] Sending connection context
0807 06:50:52.812220 (+ 251us) client_negotiation.cc:240] Negotiation successful
0807 06:50:52.812645 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":552,"threads_started":1}
I20260807 06:50:52.815590 12869 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46243
I20260807 06:50:52.815984 12869 heartbeater.cc:464] Registering TS with master...
I20260807 06:50:52.816869 12869 heartbeater.cc:511] Master 127.27.54.126:46243 requested a full tablet report, sending...
I20260807 06:50:52.818544 12195 ts_manager.cc:194] Registered new tserver with Master: ec04c9350d784c77a23e5177538baffb (127.27.54.67:33563)
I20260807 06:50:52.819550 12195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:60647
I20260807 06:50:52.823536 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:50:52.845916 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.828930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.27.54.126:46243)
0807 06:50:52.829159 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:52.829168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:52.829705 (+ 537us) server_negotiation.cc:407] Connection header received
0807 06:50:52.829870 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:52.829874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:52.829968 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:52.830057 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:52.831998 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.832822 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.834405 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.835017 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.835122 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.838550 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:52.838557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:52.838560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:52.838834 (+ 274us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:52.842911 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:52.843691 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:52.843695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:52.843697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:52.843914 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:52.844313 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:52.844316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:52.844318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:52.844614 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:52.844840 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:52.845368 (+ 528us) server_negotiation.cc:299] Negotiation successful
0807 06:50:52.845566 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:50:52.851457 12195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:50:52.860848 12195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:50:52.900516 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.875126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33681 (local address 127.0.0.1:55238)
0807 06:50:52.887328 (+ 12202us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:52.887375 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:52.887746 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:52.889063 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:52.889073 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:52.889123 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:52.889960 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.889979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.893136 (+ 3157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.893146 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.896725 (+ 3579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.896746 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.899276 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.899283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.899510 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.899562 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 06:50:52.899710 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 06:50:52.899971 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11923,"mutex_wait_us":80,"thread_start_us":11709,"threads_started":1}
I20260807 06:50:52.900928 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.876013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.27.54.66:45423)
0807 06:50:52.889014 (+ 13001us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:52.889025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:52.889057 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:50:52.889177 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:52.889184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:52.889458 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:52.889706 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:52.891010 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.894623 (+ 3613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.897834 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.899397 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.899545 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.899733 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:52.900349 (+ 616us) server_negotiation.cc:299] Negotiation successful
0807 06:50:52.900540 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12855,"thread_start_us":12713,"threads_started":1}
I20260807 06:50:52.901439 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.875311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.27.54.65:33681)
0807 06:50:52.886234 (+ 10923us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:52.886248 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:52.887522 (+ 1274us) server_negotiation.cc:407] Connection header received
0807 06:50:52.888100 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:52.888114 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:52.888440 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:52.888654 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:52.890299 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.892872 (+ 2573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.896904 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.898578 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.898785 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.899115 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:52.900835 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0807 06:50:52.901036 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10651,"thread_start_us":10955,"threads_started":1}
I20260807 06:50:52.901829 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.875687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:45423 (local address 127.0.0.1:60582)
0807 06:50:52.887898 (+ 12211us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:52.887916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:52.888054 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:52.890058 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:52.890066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:52.890104 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:52.890751 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.890769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.894875 (+ 4106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.894913 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.897614 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.897627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.899922 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.899930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.900105 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.900139 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:50:52.901234 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0807 06:50:52.901346 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12028,"mutex_wait_us":1,"thread_start_us":12494,"threads_started":1}
I20260807 06:50:52.905429 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.889159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:33563 (local address 127.0.0.1:37608)
0807 06:50:52.896617 (+ 7458us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:52.896634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:52.896822 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:52.897638 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:52.897642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:52.897681 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:52.898250 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.898262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.900380 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.900388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.902853 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:52.902869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.904513 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.904520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:52.904728 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.904767 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 06:50:52.904883 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 06:50:52.905025 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7319,"thread_start_us":7140,"threads_started":1}
I20260807 06:50:52.905496 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:52.888855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.27.54.67:33563)
0807 06:50:52.895989 (+ 7134us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:52.896003 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:52.896768 (+ 765us) server_negotiation.cc:407] Connection header received
0807 06:50:52.897067 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:52.897077 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:52.897388 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:52.897622 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:52.898503 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.900184 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.903054 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:52.904271 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:52.904446 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:52.904689 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:52.904928 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 06:50:52.905090 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6854,"thread_start_us":6568,"threads_started":1}
I20260807 06:50:52.906208 12398 tablet_service.cc:1467] Processing CreateTablet for tablet c9dae011fba3450aa89afcf96329a35c (DEFAULT_TABLE table=test-table [id=6da7156a3067484582e91dc01b8f23fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:50:52.906199 12602 tablet_service.cc:1467] Processing CreateTablet for tablet c9dae011fba3450aa89afcf96329a35c (DEFAULT_TABLE table=test-table [id=6da7156a3067484582e91dc01b8f23fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:50:52.908041 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dae011fba3450aa89afcf96329a35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:52.908499 12602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dae011fba3450aa89afcf96329a35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:52.909228 12785 tablet_service.cc:1467] Processing CreateTablet for tablet c9dae011fba3450aa89afcf96329a35c (DEFAULT_TABLE table=test-table [id=6da7156a3067484582e91dc01b8f23fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:50:52.910573 12785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dae011fba3450aa89afcf96329a35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:50:52.922827 12898 tablet_bootstrap.cc:492] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Bootstrap starting.
I20260807 06:50:52.925225 12901 tablet_bootstrap.cc:492] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Bootstrap starting.
I20260807 06:50:52.927752 12898 tablet_bootstrap.cc:654] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:52.929502 12898 log.cc:824] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Log is configured to *not* fsync() on all Append() calls
I20260807 06:50:52.929600 12901 tablet_bootstrap.cc:654] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:52.931304 12901 log.cc:824] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Log is configured to *not* fsync() on all Append() calls
I20260807 06:50:52.931438 12899 tablet_bootstrap.cc:492] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Bootstrap starting.
I20260807 06:50:52.933816 12898 tablet_bootstrap.cc:492] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: No bootstrap required, opened a new log
I20260807 06:50:52.934131 12898 ts_tablet_manager.cc:1397] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260807 06:50:52.935555 12901 tablet_bootstrap.cc:492] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: No bootstrap required, opened a new log
I20260807 06:50:52.935762 12899 tablet_bootstrap.cc:654] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Neither blocks nor log segments found. Creating new log.
I20260807 06:50:52.935894 12901 ts_tablet_manager.cc:1397] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260807 06:50:52.937330 12899 log.cc:824] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Log is configured to *not* fsync() on all Append() calls
I20260807 06:50:52.941185 12899 tablet_bootstrap.cc:492] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: No bootstrap required, opened a new log
I20260807 06:50:52.941526 12899 ts_tablet_manager.cc:1397] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260807 06:50:52.944065 12898 raft_consensus.cc:348] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:52.944506 12898 raft_consensus.cc:374] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:52.944629 12898 raft_consensus.cc:729] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec04c9350d784c77a23e5177538baffb, State: Initialized, Role: FOLLOWER
I20260807 06:50:52.945330 12898 consensus_queue.cc:260] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:52.946779 12869 heartbeater.cc:503] Master 127.27.54.126:46243 was elected leader, sending a full tablet report...
I20260807 06:50:52.947358 12898 ts_tablet_manager.cc:1428] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260807 06:50:52.947763 12901 raft_consensus.cc:348] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:52.948329 12901 raft_consensus.cc:374] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:52.948498 12901 raft_consensus.cc:729] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fb90168b74d1b9fc03d612633a9ae, State: Initialized, Role: FOLLOWER
I20260807 06:50:52.949355 12901 consensus_queue.cc:260] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:52.951413 12687 heartbeater.cc:503] Master 127.27.54.126:46243 was elected leader, sending a full tablet report...
I20260807 06:50:52.951511 12901 ts_tablet_manager.cc:1428] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 06:50:52.954931 12899 raft_consensus.cc:348] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:52.955497 12899 raft_consensus.cc:374] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:50:52.955643 12899 raft_consensus.cc:729] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d7062ea26147b4a7467a599d734837, State: Initialized, Role: FOLLOWER
I20260807 06:50:52.956400 12899 consensus_queue.cc:260] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:52.962031 12899 ts_tablet_manager.cc:1428] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
W20260807 06:50:53.041038 12871 tablet.cc:2369] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:50:53.055240 12688 tablet.cc:2369] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:50:53.208865 12487 tablet.cc:2369] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:50:53.281502 12906 raft_consensus.cc:482] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:50:53.281925 12906 raft_consensus.cc:504] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:53.283816 12906 leader_election.cc:288] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78d7062ea26147b4a7467a599d734837 (127.27.54.65:33681), 1c7fb90168b74d1b9fc03d612633a9ae (127.27.54.66:45423)
I20260807 06:50:53.291093 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.284131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33681 (local address 127.27.54.67:35857)
0807 06:50:53.284349 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:53.284375 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:53.284537 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:53.285034 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:53.285040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:53.285068 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:53.285349 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:53.285363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.286752 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.286755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:53.288831 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:53.288847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.290334 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.290341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:53.290531 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.290565 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:50:53.290699 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 06:50:53.290875 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 06:50:53.291220 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.284320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:35857 (local address 127.27.54.65:33681)
0807 06:50:53.284558 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:53.284566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:53.284596 (+ 30us) server_negotiation.cc:407] Connection header received
0807 06:50:53.284705 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:53.284714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:53.284824 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:53.284958 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:53.285509 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.286543 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.289057 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.290155 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.290318 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.290375 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:53.290768 (+ 393us) server_negotiation.cc:299] Negotiation successful
0807 06:50:53.290923 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:50:53.291664 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.284665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:45725 (local address 127.27.54.66:45423)
0807 06:50:53.284932 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:53.284938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:53.285255 (+ 317us) server_negotiation.cc:407] Connection header received
0807 06:50:53.285498 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:53.285503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:53.285590 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:53.285700 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:50:53.286274 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.287177 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.289516 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.290537 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.290737 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.290806 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:53.291279 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 06:50:53.291439 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:50:53.292416 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.284365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:45423 (local address 127.27.54.67:45725)
0807 06:50:53.285137 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0807 06:50:53.285154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:50:53.285291 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:50:53.285718 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:50:53.285720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:50:53.285741 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:50:53.286114 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:53.286122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.287325 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.287330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:53.289348 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:50:53.289359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.290870 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.290877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:50:53.291050 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.291089 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 06:50:53.291857 (+ 768us) client_negotiation.cc:240] Negotiation successful
0807 06:50:53.292007 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":450,"threads_started":1}
I20260807 06:50:53.292479 12424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dae011fba3450aa89afcf96329a35c" candidate_uuid: "ec04c9350d784c77a23e5177538baffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d7062ea26147b4a7467a599d734837" is_pre_election: true
I20260807 06:50:53.293253 12424 raft_consensus.cc:2454] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec04c9350d784c77a23e5177538baffb in term 0.
I20260807 06:50:53.293506 12626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dae011fba3450aa89afcf96329a35c" candidate_uuid: "ec04c9350d784c77a23e5177538baffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" is_pre_election: true
I20260807 06:50:53.294009 12718 leader_election.cc:302] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d7062ea26147b4a7467a599d734837, ec04c9350d784c77a23e5177538baffb; no voters:
I20260807 06:50:53.294109 12626 raft_consensus.cc:2454] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec04c9350d784c77a23e5177538baffb in term 0.
I20260807 06:50:53.294445 12906 raft_consensus.cc:2781] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:50:53.294622 12906 raft_consensus.cc:482] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:50:53.294800 12906 raft_consensus.cc:3037] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:53.297459 12906 raft_consensus.cc:504] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:53.298153 12906 leader_election.cc:288] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [CANDIDATE]: Term 1 election: Requested vote from peers 78d7062ea26147b4a7467a599d734837 (127.27.54.65:33681), 1c7fb90168b74d1b9fc03d612633a9ae (127.27.54.66:45423)
I20260807 06:50:53.298772 12424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dae011fba3450aa89afcf96329a35c" candidate_uuid: "ec04c9350d784c77a23e5177538baffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d7062ea26147b4a7467a599d734837"
I20260807 06:50:53.298806 12626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dae011fba3450aa89afcf96329a35c" candidate_uuid: "ec04c9350d784c77a23e5177538baffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fb90168b74d1b9fc03d612633a9ae"
I20260807 06:50:53.299073 12424 raft_consensus.cc:3037] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:53.299099 12626 raft_consensus.cc:3037] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:50:53.302939 12424 raft_consensus.cc:2454] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec04c9350d784c77a23e5177538baffb in term 1.
I20260807 06:50:53.302939 12626 raft_consensus.cc:2454] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec04c9350d784c77a23e5177538baffb in term 1.
I20260807 06:50:53.303484 12718 leader_election.cc:302] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d7062ea26147b4a7467a599d734837, ec04c9350d784c77a23e5177538baffb; no voters:
I20260807 06:50:53.303925 12906 raft_consensus.cc:2781] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:50:53.304462 12906 raft_consensus.cc:686] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 1 LEADER]: Becoming Leader. State: Replica: ec04c9350d784c77a23e5177538baffb, State: Running, Role: LEADER
I20260807 06:50:53.305227 12906 consensus_queue.cc:237] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } }
I20260807 06:50:53.310194 12193 catalog_manager.cc:5495] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb reported cstate change: term changed from 0 to 1, leader changed from <none> to ec04c9350d784c77a23e5177538baffb (127.27.54.67). New cstate: current_term: 1 leader_uuid: "ec04c9350d784c77a23e5177538baffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec04c9350d784c77a23e5177538baffb" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 33563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20260807 06:50:53.348279 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.342509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.27.54.67:33563)
0807 06:50:53.342845 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:53.342850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:53.342869 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:50:53.342972 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:53.342976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:53.343055 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:53.343251 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:50:53.343592 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.344337 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.345821 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.346291 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.346408 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.347083 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:50:53.347624 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:50:53.347746 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:53.347876 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 06:50:53.347955 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 06:50:53.357406 12626 raft_consensus.cc:1261] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 1 FOLLOWER]: Refusing update from remote peer ec04c9350d784c77a23e5177538baffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:50:53.357405 12424 raft_consensus.cc:1261] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 1 FOLLOWER]: Refusing update from remote peer ec04c9350d784c77a23e5177538baffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:50:53.358556 12911 consensus_queue.cc:1035] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d7062ea26147b4a7467a599d734837" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:50:53.359053 12906 consensus_queue.cc:1035] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7fb90168b74d1b9fc03d612633a9ae" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:50:53.374002 12917 mvcc.cc:205] Tried to move back new op lower bound from 7315806016940724224 to 7315806016742338560. Current Snapshot: MvccSnapshot[applied={T|T < 7315806016940724224}]
I20260807 06:50:53.374009 12915 mvcc.cc:205] Tried to move back new op lower bound from 7315806016940724224 to 7315806016742338560. Current Snapshot: MvccSnapshot[applied={T|T < 7315806016940724224}]
I20260807 06:50:53.376299 12916 mvcc.cc:205] Tried to move back new op lower bound from 7315806016940724224 to 7315806016742338560. Current Snapshot: MvccSnapshot[applied={T|T < 7315806016940724224}]
I20260807 06:50:53.389544 12193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:50:53.389849 12193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:50:53.393885 12193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af98b5287e254174825201cd8652d3e4: Sending DeleteTablet for 3 replicas of tablet c9dae011fba3450aa89afcf96329a35c
I20260807 06:50:53.395380 12398 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dae011fba3450aa89afcf96329a35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:50:53 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:55238
I20260807 06:50:53.395386 12602 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dae011fba3450aa89afcf96329a35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:50:53 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60582
I20260807 06:50:53.397315 12785 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dae011fba3450aa89afcf96329a35c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:50:53 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:37608
I20260807 06:50:53.398314 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 12306
I20260807 06:50:53.407634 12925 tablet_replica.cc:330] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: stopping tablet replica
I20260807 06:50:53.408480 12925 raft_consensus.cc:2229] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:50:53.409198 12925 raft_consensus.cc:2258] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:50:53.410300 12927 tablet_replica.cc:330] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: stopping tablet replica
I20260807 06:50:53.410925 12927 raft_consensus.cc:2229] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:50:53.411571 12927 raft_consensus.cc:2258] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:50:53.412111 12925 ts_tablet_manager.cc:1905] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:50:53.414260 12927 ts_tablet_manager.cc:1905] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:50:53.419710 12925 ts_tablet_manager.cc:1918] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:50:53.420122 12925 log.cc:1196] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/c9dae011fba3450aa89afcf96329a35c
I20260807 06:50:53.420804 12925 ts_tablet_manager.cc:1939] T c9dae011fba3450aa89afcf96329a35c P 78d7062ea26147b4a7467a599d734837: Deleting consensus metadata
I20260807 06:50:53.423121 12173 catalog_manager.cc:4841] TS 78d7062ea26147b4a7467a599d734837 (127.27.54.65:33681): tablet c9dae011fba3450aa89afcf96329a35c (table test-table [id=6da7156a3067484582e91dc01b8f23fd]) successfully deleted
I20260807 06:50:53.423704 12927 ts_tablet_manager.cc:1918] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:50:53.423974 12927 log.cc:1196] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/c9dae011fba3450aa89afcf96329a35c
I20260807 06:50:53.424629 12927 ts_tablet_manager.cc:1939] T c9dae011fba3450aa89afcf96329a35c P 1c7fb90168b74d1b9fc03d612633a9ae: Deleting consensus metadata
I20260807 06:50:53.426836 12175 catalog_manager.cc:4841] TS 1c7fb90168b74d1b9fc03d612633a9ae (127.27.54.66:45423): tablet c9dae011fba3450aa89afcf96329a35c (table test-table [id=6da7156a3067484582e91dc01b8f23fd]) successfully deleted
I20260807 06:50:53.427704 12928 tablet_replica.cc:330] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: stopping tablet replica
I20260807 06:50:53.428259 12928 raft_consensus.cc:2229] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:50:53.429307 12928 raft_consensus.cc:2258] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:50:53.434494 12928 ts_tablet_manager.cc:1905] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 06:50:53 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:50:53.443087 12928 ts_tablet_manager.cc:1918] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:50:53.443423 12928 log.cc:1196] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/c9dae011fba3450aa89afcf96329a35c
I20260807 06:50:53.444164 12928 ts_tablet_manager.cc:1939] T c9dae011fba3450aa89afcf96329a35c P ec04c9350d784c77a23e5177538baffb: Deleting consensus metadata
I20260807 06:50:53.446221 12173 catalog_manager.cc:4841] TS ec04c9350d784c77a23e5177538baffb (127.27.54.67:33563): tablet c9dae011fba3450aa89afcf96329a35c (table test-table [id=6da7156a3067484582e91dc01b8f23fd]) successfully deleted
I20260807 06:50:53.447492 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.400427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.27.54.65:33681)
0807 06:50:53.401259 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:53.401269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:53.424578 (+ 23309us) server_negotiation.cc:407] Connection header received
0807 06:50:53.424958 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:53.424966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:53.425061 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:53.425157 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:53.427993 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.429483 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.431760 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.432482 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.432592 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.439097 (+ 6505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:53.439120 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:53.439140 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:53.439191 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:53.443615 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:53.444692 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:53.444700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:53.444702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:53.444858 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:53.445323 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:53.445327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:53.445329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:53.445741 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:53.446099 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:53.446852 (+ 753us) server_negotiation.cc:299] Negotiation successful
0807 06:50:53.447097 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626}
I20260807 06:50:53.448231 12481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:53.548384 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 12306
I20260807 06:50:53.575212 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 12492
Aug 07 06:50:53 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:50:53.594142 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.576045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.27.54.66:45423)
0807 06:50:53.576257 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:53.576263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:53.576290 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:50:53.576486 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:53.576492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:53.576587 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:53.576715 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:53.578044 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.579288 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.580813 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.581493 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.581637 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.586599 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:53.586623 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:53.586644 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:53.586763 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:53.590595 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:53.591674 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:53.591691 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:53.591693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:53.591863 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:53.592341 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:53.592347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:53.592351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:53.592781 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:53.593096 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:53.593455 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 06:50:53.593694 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 06:50:53.594992 12682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:53.688441 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 12492
I20260807 06:50:53.715081 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 12693
Aug 07 06:50:53 dist-test-slave-3kk6 krb5kdc[11502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:50:53.732511 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:50:53.715933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.27.54.67:33563)
0807 06:50:53.716136 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 06:50:53.716143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:50:53.716236 (+ 93us) server_negotiation.cc:407] Connection header received
0807 06:50:53.716480 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:50:53.716484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:50:53.716552 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:50:53.716676 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:50:53.718074 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.718988 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.720388 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:50:53.720849 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:50:53.720998 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:50:53.725903 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:50:53.725919 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:50:53.725937 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:50:53.725981 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:50:53.729261 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:53.730116 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:53.730125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:53.730127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:53.730284 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:50:53.730774 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:50:53.730781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:50:53.730788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:50:53.731157 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:50:53.731528 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 06:50:53.731864 (+ 336us) server_negotiation.cc:299] Negotiation successful
0807 06:50:53.732117 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 06:50:53.733213 12862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:53.868495 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 12693
I20260807 06:50:53.900398 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 12149
I20260807 06:50:53.901790 12237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:50:54.019626 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 12149
I20260807 06:50:54.057446 27865 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 06:50:54.404271 27865 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 06:50:54.404626 27865 mini_ranger.cc:66] Stopping Ranger...
I20260807 06:50:54.761862 27865 mini_ranger.cc:68] Stopped Ranger
2026-08-07 06:50:54.762 UTC [11542] LOG: received smart shutdown request
2026-08-07 06:50:54.767 UTC [11542] LOG: background worker "logical replication launcher" (PID 11555) exited with exit code 1
2026-08-07 06:50:54.767 UTC [11547] LOG: shutting down
2026-08-07 06:50:54.948 UTC [11542] LOG: database system is shut down
2026-08-07T06:50:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53977 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:50:55 dist-test-slave-3kk6 krb5kdc[12948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:50:55 dist-test-slave-3kk6 krb5kdc[12948](info): set up 2 sockets
Aug 07 06:50:55 dist-test-slave-3kk6 krb5kdc[12948](info): commencing operation
krb5kdc: starting...
W20260807 06:50:56.436993 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:50:56 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:50:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/rangeradmin_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/rangeradmin_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/rangerlookup_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/rangerlookup_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
I20260807 06:50:56.894858 27865 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHOwvC2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/postgres -l logfile start
2026-08-07 06:50:59.605 UTC [12987] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 06:50:59.605 UTC [12987] LOG: listening on IPv4 address "127.27.54.84", port 43349
2026-08-07 06:50:59.609 UTC [12987] LOG: listening on Unix socket "/tmp/.s.PGSQL.43349"
2026-08-07 06:50:59.635 UTC [12991] LOG: database system was shut down at 2026-08-07 06:50:58 UTC
2026-08-07 06:50:59.639 UTC [12987] LOG: database system is ready to accept connections
I20260807 06:51:00.816823 27865 mini_postgres.cc:96] Postgres bound to 43349
2026-08-07 06:51:00.844 UTC [13007] FATAL: database "slave" does not exist
127.27.54.84:43349 - accepting connections
I20260807 06:51:00.845434 27865 mini_ranger.cc:161] Starting Ranger...
I20260807 06:51:00.882874 27865 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 06:51:01.451642 27865 mini_ranger.cc:87] Created ranger Postgres database
I20260807 06:51:01.451786 27865 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-admin
2026-08-07 06:51:01,787 [I] DB FLAVOR :POSTGRES
2026-08-07 06:51:01,787 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:51:01,787 [I] Checking connection..
2026-08-07 06:51:01,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 06:51:02,061 [I] Checking connection passed.
2026-08-07 06:51:02,062 [I] --------- Verifying version history table ---------
2026-08-07 06:51:02,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:51:02,328 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:51:02,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:51:02,602 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 06:51:02,602 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 06:51:02,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 06:51:02.876 UTC [13088] WARNING: there is no transaction in progress
2026-08-07 06:51:02,886 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 06:51:02,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 06:51:03,179 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 06:51:03,179 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:51:03,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 06:51:03,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 06:51:03,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3kk6.c.gcp-upstream.internal','N') ;"
2026-08-07 06:51:03,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:51:04,233 [I] Table x_portal_user does not exist in database ranger
2026-08-07 06:51:04,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:51:04,513 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 06:51:04,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:51:04,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:51:05,048 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 06:51:05,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 06:51:06.126 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.144 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.162 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.180 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.200 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.219 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.281 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.289 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.304 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.316 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.328 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.344 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.355 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.364 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.372 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:06.380 UTC [13236] WARNING: there is no transaction in progress
2026-08-07 06:51:07,108 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 06:51:07,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 06:51:07,378 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 06:51:07,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 06:51:07,654 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 06:51:07,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 06:51:07,916 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 06:51:07,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 06:51:08,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 06:51:08,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3kk6.c.gcp-upstream.internal';"
2026-08-07 06:51:08,746 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 06:51:08,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 06:51:09,026 [I] DB_PATCHES have already been applied
I20260807 06:51:09.033257 27865 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHOwvC2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 06:51:09.033366 27865 mini_ranger.cc:193] Using host: 127.27.54.84
I20260807 06:51:09.057484 27865 mini_ranger.cc:227] Ranger admin URL: http://127.27.54.84:39821
Aug 07, 2026 6:51:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:51:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:51:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 6:51:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 6:51:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 6:51:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 6:51:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 06:51:10.484920 27865 mini_ranger.cc:160] Time spent starting Ranger: real 9.640s user 0.001s sys 0.075s
Aug 07, 2026 6:51:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 6:51:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 6:51:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 6:51:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 6:51:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 06:51:34.103331 27865 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.54.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.54.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/rangerkms_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.54.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/rangerkms_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.54.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.54.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.54.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/HTTP_127.27.54.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 06:51:34 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 06:51:34.341153 27865 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 06:51:34.371290 27865 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 06:51:35.307282 27865 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 06:51:35.307435 27865 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms
I20260807 06:51:35.307482 27865 mini_ranger_kms.cc:227] Using postgres at 127.27.54.84:43349
2026-08-07 06:51:35,793 [I] DB FLAVOR :POSTGRES
2026-08-07 06:51:35,793 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 06:51:35,793 [I] Checking connection
2026-08-07 06:51:35,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 06:51:36,075 [I] connection success
2026-08-07 06:51:36,075 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 06:51:36,075 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 06:51:36,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 06:51:36,354 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 06:51:36,355 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 06:51:36,355 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 06:51:36,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.54.84:43349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 06:51:36,636 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 06:51:37.152915 27865 mini_ranger_kms.cc:310] Created kms service
I20260807 06:51:37.364858 27865 mini_ranger_kms.cc:326] Created kudu user
I20260807 06:51:37.415522 27865 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 06:51:37 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:37.897974 27865 mini_ranger_kms.cc:384] Added ranger policy
I20260807 06:51:37.898321 27865 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHOwvC2/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHOwvC2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 06:51:37.898379 27865 mini_ranger_kms.cc:240] Using host: 127.27.54.84
I20260807 06:51:37.923389 27865 mini_ranger_kms.cc:276] Ranger KMS PID: 13478
I20260807 06:51:37.923513 27865 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.54.84:35007
Aug 07, 2026 6:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 6:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 6:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 6:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 6:51:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 6:51:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 06:51:40 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786085500, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.54.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:40.714740 27865 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.374s user 0.064s sys 0.449s
I20260807 06:51:40.715024 27865 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 06:51:40.715215 27865 mini_ranger_kms.cc:401] 127.27.54.84:35007/kms/v1/keys
Aug 07 06:51:40 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085500, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.54.84@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
Aug 07 06:51:41 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085500, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.54.84@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
Aug 07 06:51:43 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:51:43.776927 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40407
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40407
--ranger_config_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:51:44.141499 13545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:44.141893 13545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:44.142055 13545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:44.149590 13545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:51:44.149765 13545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:44.149924 13545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:44.150055 13545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:51:44.150143 13545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:51:44.160946 13545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.54.84:35007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40407
--ranger_config_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:44.162387 13545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:44.163837 13545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:44.181790 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:44.182036 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:44.182251 13557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:44.182873 13545 server_base.cc:1034] running on GCE node
I20260807 06:51:44.183725 13545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:44.185329 13545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:44.186555 13545 hybrid_clock.cc:648] HybridClock initialized: now 1786085504186517 us; error 37 us; skew 500 ppm
I20260807 06:51:44.186970 13545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:44 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:44.194236 13545 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:51:44.196163 13545 webserver.cc:460] Webserver started at http://127.27.54.126:35413/ using document root <none> and password file <none>
I20260807 06:51:44.196864 13545 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:44.197010 13545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:44.197383 13545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:51:44 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:44.265055 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5"
format_stamp: "Formatted at 2026-08-07 06:51:44 on dist-test-slave-3kk6"
server_key: "fd9a24054f9b3cb8511d8b2783e45e96"
server_key_iv: "82198a3e2b14c56bb5a6248bbc7123d6"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:44.266036 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5"
format_stamp: "Formatted at 2026-08-07 06:51:44 on dist-test-slave-3kk6"
server_key: "fd9a24054f9b3cb8511d8b2783e45e96"
server_key_iv: "82198a3e2b14c56bb5a6248bbc7123d6"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:44.272032 13545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 06:51:44.293231 13564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:44.295157 13545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 06:51:44.295424 13545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5"
format_stamp: "Formatted at 2026-08-07 06:51:44 on dist-test-slave-3kk6"
server_key: "fd9a24054f9b3cb8511d8b2783e45e96"
server_key_iv: "82198a3e2b14c56bb5a6248bbc7123d6"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:44.295667 13545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:44.317723 13545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:44.322844 13545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:44.323276 13545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:44.346990 13545 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40407
I20260807 06:51:44.346990 13636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40407 every 8 connection(s)
I20260807 06:51:44.348685 13545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:51:44.353046 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 13545
I20260807 06:51:44.353327 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:51:44.354068 13637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:44.354104 27865 external_mini_cluster.cc:1428] Setting key d7b00e2f65b116927b37a10da9ce74bc
I20260807 06:51:44.370821 13637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Bootstrap starting.
I20260807 06:51:44.375389 13637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:44.376895 13637 log.cc:824] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:44.380790 13637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: No bootstrap required, opened a new log
Aug 07 06:51:44 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:44.392625 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:44.356695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.27.54.126:40407)
0807 06:51:44.368425 (+ 11730us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:44.368455 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:44.369540 (+ 1085us) server_negotiation.cc:407] Connection header received
0807 06:51:44.372220 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:44.372249 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:44.372710 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:44.373196 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:44.374944 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:44.376548 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:44.378612 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:44.379412 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:44.379642 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:44.384248 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:44.384259 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:44.384276 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:44.384321 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:44.387913 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:44.388757 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:44.388764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:44.388771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:44.388947 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:44.389415 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:44.389419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:44.389422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:44.390083 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:44.390449 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:44.390992 (+ 543us) server_negotiation.cc:299] Negotiation successful
0807 06:51:44.391507 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11301,"thread_start_us":11644,"threads_started":1}
I20260807 06:51:44.392275 13637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:44.392876 13637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:44.393034 13637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f0e5792a35b4bb9b3f2a80f03db37b5, State: Initialized, Role: FOLLOWER
I20260807 06:51:44.393786 13637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:44.394236 13637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:51:44.394476 13637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:51:44.394897 13637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:44.398154 13637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:44.398789 13637 leader_election.cc:302] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f0e5792a35b4bb9b3f2a80f03db37b5; no voters:
I20260807 06:51:44.399865 13637 leader_election.cc:288] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:51:44.400029 13645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:51:44.402244 13637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:51:44.407073 13645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 LEADER]: Becoming Leader. State: Replica: 6f0e5792a35b4bb9b3f2a80f03db37b5, State: Running, Role: LEADER
I20260807 06:51:44.407819 13645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:44.412708 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } } }
I20260807 06:51:44.413187 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:44.413782 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f0e5792a35b4bb9b3f2a80f03db37b5. Latest consensus state: current_term: 1 leader_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } } }
I20260807 06:51:44.414134 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:44.436473 13653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:51:44.450006 13653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:51:44.458639 13637 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 06:51:44.473344 13653 catalog_manager.cc:1347] Generated new cluster ID: 63a24e15fb224c439b574defa8eb810c
I20260807 06:51:44.473658 13653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:51:44.512617 13653 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:51:44.515393 13653 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:51:44.526350 13653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Generated new TSK 0
I20260807 06:51:44.527837 13653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:51:45.415678 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40407
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:51:45 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786085505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 06:51:45.895218 13703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:45.895767 13703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:45.895956 13703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:45.906255 13703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:45.906457 13703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:45.906700 13703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:51:45.917292 13703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.54.84:35007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:45.918880 13703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:45.920267 13703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:45.947948 13714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:45.952513 13715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:45.964389 13703 server_base.cc:1034] running on GCE node
W20260807 06:51:45.964991 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:45.965919 13703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:45.967190 13703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:45.968470 13703 hybrid_clock.cc:648] HybridClock initialized: now 1786085505968262 us; error 207 us; skew 500 ppm
I20260807 06:51:45.968863 13703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:45 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:45.972733 13703 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:51:45.977423 13703 webserver.cc:460] Webserver started at http://127.27.54.65:41223/ using document root <none> and password file <none>
I20260807 06:51:45.978353 13703 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:45.978528 13703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:45.979023 13703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:51:45 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:46.010227 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "925c1fd750c247988a63f77387a3311d"
format_stamp: "Formatted at 2026-08-07 06:51:45 on dist-test-slave-3kk6"
server_key: "ada8cfd27a7f83a6da6f1e078c03c0a5"
server_key_iv: "c18fe14e1765b6c54824144db9b9640f"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:46.013291 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "925c1fd750c247988a63f77387a3311d"
format_stamp: "Formatted at 2026-08-07 06:51:45 on dist-test-slave-3kk6"
server_key: "ada8cfd27a7f83a6da6f1e078c03c0a5"
server_key_iv: "c18fe14e1765b6c54824144db9b9640f"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:46.028003 13703 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.009s sys 0.003s
I20260807 06:51:46.063405 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:46.065801 13703 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.011s sys 0.008s
I20260807 06:51:46.066104 13703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "925c1fd750c247988a63f77387a3311d"
format_stamp: "Formatted at 2026-08-07 06:51:45 on dist-test-slave-3kk6"
server_key: "ada8cfd27a7f83a6da6f1e078c03c0a5"
server_key_iv: "c18fe14e1765b6c54824144db9b9640f"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:46.066496 13703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:46.121184 13703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:46.126761 13703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:46.127169 13703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:46.138743 13703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:51:46.141518 13703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:51:46.141685 13703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 06:51:46.141839 13703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:51:46.141898 13703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 06:51:46 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:46.259135 13703 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:46683
I20260807 06:51:46.259240 13886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:46683 every 8 connection(s)
I20260807 06:51:46.260577 13703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:51:46.268303 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 13703
I20260807 06:51:46.268565 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:51:46.269418 27865 external_mini_cluster.cc:1428] Setting key 8782e5f85055a98cf045342da629ea8f
Aug 07 06:51:46 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:46.284379 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:46.264112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:54803 (local address 127.27.54.126:40407)
0807 06:51:46.264663 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:46.264669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:46.265790 (+ 1121us) server_negotiation.cc:407] Connection header received
0807 06:51:46.268676 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:46.268683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:46.268803 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:46.268948 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:46.271122 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:46.272209 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:46.274017 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:46.274441 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:46.274584 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:46.278187 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:46.278192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:46.278196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:46.278228 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:46.281020 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:46.281822 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:46.281827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:46.281829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:46.281969 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:46.282457 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:46.282462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:46.282464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:46.282813 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:46.282974 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:46.283739 (+ 765us) server_negotiation.cc:299] Negotiation successful
0807 06:51:46.283979 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
I20260807 06:51:46.285459 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:46.264668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40407 (local address 127.27.54.65:54803)
0807 06:51:46.265562 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:46.265620 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:46.268180 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:46.269269 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:46.269282 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:46.269979 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:46.270919 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:46.270949 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:46.272381 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:46.272386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:46.273847 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:46.273861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:46.274756 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:46.274766 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:46.274996 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:46.275759 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:46.275782 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:46.277942 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:46.281190 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:46.281200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:46.281218 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:46.281679 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:46.282093 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:46.282098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:46.282102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:46.282314 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:46.282948 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:46.282956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:46.283326 (+ 370us) client_negotiation.cc:769] Sending connection context
0807 06:51:46.283624 (+ 298us) client_negotiation.cc:240] Negotiation successful
0807 06:51:46.284194 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":321,"threads_started":1}
I20260807 06:51:46.287276 13887 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40407
I20260807 06:51:46.287708 13887 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:46.288762 13887 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:46.291616 13592 ts_manager.cc:194] Registered new tserver with Master: 925c1fd750c247988a63f77387a3311d (127.27.54.65:46683)
I20260807 06:51:46.293814 13592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:54803
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:51:46.355489 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40407
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:51:46 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
W20260807 06:51:46.860327 13893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:46.860764 13893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:46.860937 13893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:46.868527 13893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:46.868721 13893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:46.868927 13893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:51:46.879781 13893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.54.84:35007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:46.881357 13893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:46.882874 13893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:46.901983 13914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:46.907564 13912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:46.909080 13916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:46.910202 13893 server_base.cc:1034] running on GCE node
I20260807 06:51:46.911062 13893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:46.912469 13893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:46.913741 13893 hybrid_clock.cc:648] HybridClock initialized: now 1786085506913666 us; error 69 us; skew 500 ppm
I20260807 06:51:46.914101 13893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:46 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:46.917682 13893 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:51:46.919724 13893 webserver.cc:460] Webserver started at http://127.27.54.66:36567/ using document root <none> and password file <none>
I20260807 06:51:46.920578 13893 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:46.920749 13893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:46.921099 13893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:51:46 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:46.943218 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "b65312f2f09844a39d04095ad4f5cd1a"
format_stamp: "Formatted at 2026-08-07 06:51:46 on dist-test-slave-3kk6"
server_key: "fc461f01284847fe6a4a4534767bffe7"
server_key_iv: "8533678039334fb0910008b7e8e376b9"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:46.944590 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "b65312f2f09844a39d04095ad4f5cd1a"
format_stamp: "Formatted at 2026-08-07 06:51:46 on dist-test-slave-3kk6"
server_key: "fc461f01284847fe6a4a4534767bffe7"
server_key_iv: "8533678039334fb0910008b7e8e376b9"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:46.960858 13893 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.010s sys 0.000s
I20260807 06:51:47.013542 13928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:47.016201 13893 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.005s sys 0.012s
I20260807 06:51:47.016628 13893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "b65312f2f09844a39d04095ad4f5cd1a"
format_stamp: "Formatted at 2026-08-07 06:51:46 on dist-test-slave-3kk6"
server_key: "fc461f01284847fe6a4a4534767bffe7"
server_key_iv: "8533678039334fb0910008b7e8e376b9"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:47.017112 13893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:47.100505 13893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:47.107163 13893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:47.107625 13893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:47.109056 13893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:51:47.111184 13893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:51:47.111338 13893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 06:51:47.111506 13893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:51:47.111598 13893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:47.154644 13893 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:43743
I20260807 06:51:47.154718 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:43743 every 8 connection(s)
I20260807 06:51:47.156795 13893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:51:47.157729 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 13893
I20260807 06:51:47.157910 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:51:47.158473 27865 external_mini_cluster.cc:1428] Setting key d66c352b02626dd440606f1e5c51d5cd
Aug 07 06:51:47 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085506, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:47.211905 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:47.172589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:43485 (local address 127.27.54.126:40407)
0807 06:51:47.189138 (+ 16549us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:47.189145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:47.189172 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:51:47.189262 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:47.189270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:47.189373 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:47.189516 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:47.192393 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.193687 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.196698 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.197460 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.197594 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:47.203907 (+ 6313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:47.203913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:47.203916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:47.203955 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:47.207541 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:47.208853 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:47.208867 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:47.208869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:47.209012 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:47.209627 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:47.209632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:47.209638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:47.209952 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:47.210138 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:47.211218 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0807 06:51:47.211471 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16378,"thread_start_us":17077,"threads_started":1}
I20260807 06:51:47.213335 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:47.173429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40407 (local address 127.27.54.66:43485)
0807 06:51:47.184602 (+ 11173us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:47.184689 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:47.187449 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:47.190159 (+ 2710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:47.190178 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:47.190980 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:47.192074 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:47.192114 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.193965 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.193974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:47.196462 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:47.196484 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.198060 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.198071 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:47.198454 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:47.199469 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:47.199508 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:47.203585 (+ 4077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:47.207751 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:47.207765 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:47.207790 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:47.208693 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:47.209149 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:47.209156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:47.209162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:47.209489 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:47.210102 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:47.210111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:47.210717 (+ 606us) client_negotiation.cc:769] Sending connection context
0807 06:51:47.211108 (+ 391us) client_negotiation.cc:240] Negotiation successful
0807 06:51:47.211774 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10760,"thread_start_us":14428,"threads_started":1}
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
I20260807 06:51:47.215463 14076 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40407
I20260807 06:51:47.215837 14076 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:47.216625 14076 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:47.218400 13592 ts_manager.cc:194] Registered new tserver with Master: b65312f2f09844a39d04095ad4f5cd1a (127.27.54.66:43743)
I20260807 06:51:47.219369 13592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:43485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:51:47.257869 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40407
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:51:47.298727 13887 heartbeater.cc:503] Master 127.27.54.126:40407 was elected leader, sending a full tablet report...
W20260807 06:51:47.641629 14087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:47.642099 14087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:47.642273 14087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:47.650362 14087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:47.650514 14087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:47.650760 14087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:51:47.661906 14087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.54.84:35007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:47.663489 14087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:47.664994 14087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:47.679139 14096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:47.679117 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:47.679368 14094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:47.680292 14087 server_base.cc:1034] running on GCE node
I20260807 06:51:47.680998 14087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:47.682061 14087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:47.683302 14087 hybrid_clock.cc:648] HybridClock initialized: now 1786085507683253 us; error 48 us; skew 500 ppm
I20260807 06:51:47.683660 14087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:47 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:47.691179 14087 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:51:47.693077 14087 webserver.cc:460] Webserver started at http://127.27.54.67:39003/ using document root <none> and password file <none>
I20260807 06:51:47.693910 14087 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:47.694056 14087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:47.694507 14087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 06:51:47 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:47.714965 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "36ab11e846184a9990fedaa51fb2f2ef"
format_stamp: "Formatted at 2026-08-07 06:51:47 on dist-test-slave-3kk6"
server_key: "9b4cc843491ba1e15b3d69b018621748"
server_key_iv: "e7c063caf6d7781a981bac5d51e5c5b5"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:47.716389 14087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "36ab11e846184a9990fedaa51fb2f2ef"
format_stamp: "Formatted at 2026-08-07 06:51:47 on dist-test-slave-3kk6"
server_key: "9b4cc843491ba1e15b3d69b018621748"
server_key_iv: "e7c063caf6d7781a981bac5d51e5c5b5"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:47.724432 14087 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260807 06:51:47.753767 14110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:47.755905 14087 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.003s
I20260807 06:51:47.756192 14087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "36ab11e846184a9990fedaa51fb2f2ef"
format_stamp: "Formatted at 2026-08-07 06:51:47 on dist-test-slave-3kk6"
server_key: "9b4cc843491ba1e15b3d69b018621748"
server_key_iv: "e7c063caf6d7781a981bac5d51e5c5b5"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:47.756493 14087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:47.784289 14087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:47.790421 14087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:47.790863 14087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:47.792356 14087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:51:47.794405 14087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:51:47.794499 14087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:47.794589 14087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:51:47.794637 14087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:47.828699 14087 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:41329
I20260807 06:51:47.828801 14260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:41329 every 8 connection(s)
I20260807 06:51:47.830111 14087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:51:47.831517 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 14087
I20260807 06:51:47.831714 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:51:47.832144 27865 external_mini_cluster.cc:1428] Setting key b166e26963318bcb7117439a32483d62
Aug 07 06:51:47 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:47.851925 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:47.833142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:46429 (local address 127.27.54.126:40407)
0807 06:51:47.833647 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:47.833653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:47.834832 (+ 1179us) server_negotiation.cc:407] Connection header received
0807 06:51:47.837051 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:47.837056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:47.837148 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:47.837280 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:47.838856 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.839608 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.841180 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.841626 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.841772 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:47.845388 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:47.845393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:47.845397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:47.845453 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:47.848458 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:47.849366 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:47.849370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:47.849373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:47.849504 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:47.850013 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:47.850017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:47.850019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:47.850325 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:47.850490 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:47.851290 (+ 800us) server_negotiation.cc:299] Negotiation successful
0807 06:51:47.851535 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260807 06:51:47.852900 14265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:47.833644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40407 (local address 127.27.54.67:46429)
0807 06:51:47.834609 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:47.834691 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:47.836782 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:47.837527 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:47.837536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:47.837977 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:47.838629 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:47.838692 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.839772 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.839784 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:47.841020 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:47.841032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.841863 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.841872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:47.842113 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:47.842920 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:47.842941 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:47.845147 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:47.848598 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:47.848608 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:47.848627 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:47.849225 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:47.849621 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:47.849626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:47.849630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:47.849879 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:47.850471 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:47.850480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:47.850899 (+ 419us) client_negotiation.cc:769] Sending connection context
0807 06:51:47.851184 (+ 285us) client_negotiation.cc:240] Negotiation successful
0807 06:51:47.851746 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":382,"threads_started":1}
I20260807 06:51:47.854252 14261 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40407
I20260807 06:51:47.854611 14261 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:47.855386 14261 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:47.856935 13592 ts_manager.cc:194] Registered new tserver with Master: 36ab11e846184a9990fedaa51fb2f2ef (127.27.54.67:41329)
I20260807 06:51:47.857820 13592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:46429
I20260807 06:51:47.861546 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:51:47.863391 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 13703
Aug 07 06:51:47 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:51:47.881261 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:47.864520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.27.54.65:46683)
0807 06:51:47.865197 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:47.865204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:47.865235 (+ 31us) server_negotiation.cc:407] Connection header received
0807 06:51:47.865296 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:47.865302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:47.865506 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:47.865629 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:47.867225 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.868255 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.869853 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.870413 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.870515 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:47.875449 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:47.875462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:47.875477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:47.875520 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:47.878430 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:47.879202 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:47.879210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:47.879212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:47.879369 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:47.879772 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:47.879775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:47.879778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:47.880116 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:47.880363 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:47.880731 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 06:51:47.880954 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":351,"threads_started":1}
I20260807 06:51:47.882817 13881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:47.968117 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 13703
I20260807 06:51:47.989006 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 13893
Aug 07 06:51:47 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:51:48.007274 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:47.990119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.27.54.66:43743)
0807 06:51:47.990788 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:47.990795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:47.990821 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:51:47.990887 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:47.990894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:47.991120 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:47.991253 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:47.992580 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.993605 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.995134 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:47.995703 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:47.995811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:48.000852 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:48.000865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:48.000879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:48.000927 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:48.004150 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:48.005012 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:48.005019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:48.005022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:48.005169 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:48.005515 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:48.005519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:48.005521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:48.005890 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:48.006120 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:48.006478 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 06:51:48.006830 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":288,"threads_started":1}
I20260807 06:51:48.008630 14069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:48.092875 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 13893
I20260807 06:51:48.120661 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 14087
Aug 07 06:51:48 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:51:48.138192 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:48.121688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.27.54.67:41329)
0807 06:51:48.122334 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:48.122341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:48.122370 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:51:48.122437 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:48.122443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:48.122724 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:48.122916 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:48.124362 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:48.125451 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:48.126889 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:48.127489 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:48.127644 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:48.132305 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:48.132317 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:48.132333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:48.132370 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:48.135441 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:48.136205 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:48.136214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:48.136216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:48.136364 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:48.136740 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:48.136743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:48.136745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:48.137104 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:48.137357 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:48.137642 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 06:51:48.137887 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":321,"threads_started":1}
I20260807 06:51:48.139696 14251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:48.226464 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 14087
I20260807 06:51:48.250341 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 13545
I20260807 06:51:48.251791 13632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:48.362221 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 13545
I20260807 06:51:48.395663 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40407
--webserver_interface=127.27.54.126
--webserver_port=35413
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40407
--ranger_config_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:51:48.755858 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:48.756242 14276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:48.756423 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:48.764003 14276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:51:48.764189 14276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:48.764286 14276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:48.764406 14276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:51:48.764632 14276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:51:48.775504 14276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40407
--ranger_config_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=35413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:48.777088 14276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:48.778563 14276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:48.791182 14284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:48.795691 14288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:48.795828 14276 server_base.cc:1034] running on GCE node
W20260807 06:51:48.796069 14285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:48.797031 14276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:48.798595 14276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:48.799867 14276 hybrid_clock.cc:648] HybridClock initialized: now 1786085508799827 us; error 50 us; skew 500 ppm
I20260807 06:51:48.800280 14276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:48 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:48.803928 14276 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:51:48.805627 14276 webserver.cc:460] Webserver started at http://127.27.54.126:35413/ using document root <none> and password file <none>
I20260807 06:51:48.806360 14276 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:48.806527 14276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:51:48 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:48.892218 14276 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 06:51:48.895594 14299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:48.897193 14276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:51:48.897378 14276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5"
format_stamp: "Formatted at 2026-08-07 06:51:44 on dist-test-slave-3kk6"
server_key: "fd9a24054f9b3cb8511d8b2783e45e96"
server_key_iv: "82198a3e2b14c56bb5a6248bbc7123d6"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:48.898393 14276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:48.917603 14276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:48.922151 14276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:48.922447 14276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:48.942747 14276 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40407
I20260807 06:51:48.942800 14374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40407 every 8 connection(s)
I20260807 06:51:48.944468 14276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:51:48.950608 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 14276
I20260807 06:51:48.951916 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:46683
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:40407
--webserver_port=41223
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:51:48.953217 14375 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:51:48.959328 14375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Bootstrap starting.
I20260807 06:51:48.982542 14375 log.cc:824] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:48.993144 14375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:51:48.993801 14375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Bootstrap complete.
I20260807 06:51:49.004757 14375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:49.005793 14375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f0e5792a35b4bb9b3f2a80f03db37b5, State: Initialized, Role: FOLLOWER
I20260807 06:51:49.006537 14375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:49.006893 14375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:51:49.007056 14375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:51:49.007301 14375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:51:49.011826 14375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:49.012413 14375 leader_election.cc:302] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f0e5792a35b4bb9b3f2a80f03db37b5; no voters:
I20260807 06:51:49.013214 14375 leader_election.cc:288] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:51:49.013384 14383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:51:49.014840 14383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [term 2 LEADER]: Becoming Leader. State: Replica: 6f0e5792a35b4bb9b3f2a80f03db37b5, State: Running, Role: LEADER
I20260807 06:51:49.015568 14383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } }
I20260807 06:51:49.015916 14375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:51:49.018362 14385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f0e5792a35b4bb9b3f2a80f03db37b5. Latest consensus state: current_term: 2 leader_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } } }
I20260807 06:51:49.018798 14385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:49.019446 14384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f0e5792a35b4bb9b3f2a80f03db37b5" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40407 } } }
I20260807 06:51:49.019742 14384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:49.028404 14389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:51:49.036574 14389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:51:49.038899 14389 catalog_manager.cc:1259] Loaded cluster ID: 63a24e15fb224c439b574defa8eb810c
I20260807 06:51:49.039150 14389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:51:49.044821 14389 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:51:49.045369 14375 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 06:51:49.046634 14389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Loaded TSK: 0
I20260807 06:51:49.049015 14389 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:51:49.438400 14380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:49.438891 14380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:49.439148 14380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:49.451287 14380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:49.451459 14380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:49.451743 14380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:51:49.468374 14380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:46683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=41223
--tserver_master_addrs=127.27.54.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:49.470523 14380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:49.472548 14380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:49.510949 14425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:49.515657 14380 server_base.cc:1034] running on GCE node
W20260807 06:51:49.516095 14428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:49.516654 14422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:49.517686 14380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:49.522106 14380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:49.523481 14380 hybrid_clock.cc:648] HybridClock initialized: now 1786085509523383 us; error 95 us; skew 500 ppm
I20260807 06:51:49.523978 14380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:49 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:49.535239 14380 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:51:49.538777 14380 webserver.cc:460] Webserver started at http://127.27.54.65:41223/ using document root <none> and password file <none>
I20260807 06:51:49.539835 14380 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:49.540040 14380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:51:49 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:49.591948 14380 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.006s sys 0.001s
I20260807 06:51:49.598774 14436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:49.601037 14380 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260807 06:51:49.601383 14380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "925c1fd750c247988a63f77387a3311d"
format_stamp: "Formatted at 2026-08-07 06:51:45 on dist-test-slave-3kk6"
server_key: "ada8cfd27a7f83a6da6f1e078c03c0a5"
server_key_iv: "c18fe14e1765b6c54824144db9b9640f"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:49.603024 14380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:49.727384 14380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:49.733487 14380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:49.733923 14380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:49.735389 14380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:51:49.737687 14380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:51:49.737851 14380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 06:51:49.738023 14380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:51:49.738186 14380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:49.834487 14380 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:46683
I20260807 06:51:49.834533 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:46683 every 8 connection(s)
I20260807 06:51:49.836280 14380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:51:49.841725 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 14380
I20260807 06:51:49.842710 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:43743
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:40407
--webserver_port=36567
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:51:49 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:49.878463 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:49.842925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40407 (local address 127.27.54.65:44647)
0807 06:51:49.845488 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:49.845551 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:49.848177 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:49.854728 (+ 6551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:49.854738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:49.855272 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:49.855886 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:49.855918 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:49.857819 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:49.857825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:49.860011 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:49.860035 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:49.862156 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:49.862166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:49.862470 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:49.863463 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:49.863498 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:49.866678 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:49.872136 (+ 5458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:49.872148 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:49.872168 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:49.872952 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:49.873645 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:49.873651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:49.873654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:49.873978 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:49.875299 (+ 1321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:49.875307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:49.875792 (+ 485us) client_negotiation.cc:769] Sending connection context
0807 06:51:49.876151 (+ 359us) client_negotiation.cc:240] Negotiation successful
0807 06:51:49.876853 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"thread_start_us":3582,"threads_started":1}
I20260807 06:51:49.881821 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:49.851317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:44647 (local address 127.27.54.126:40407)
0807 06:51:49.852246 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:49.852261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:49.852337 (+ 76us) server_negotiation.cc:407] Connection header received
0807 06:51:49.852913 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:49.852938 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:49.853503 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:49.854028 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:49.856161 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:49.857566 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:49.860301 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:49.861430 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:49.861670 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:49.867146 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:49.867161 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:49.867184 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:49.867241 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:49.871841 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:49.873197 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:49.873208 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:49.873217 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:49.873460 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:49.874150 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:49.874155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:49.874158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:49.874969 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:49.875197 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:49.879617 (+ 4420us) server_negotiation.cc:299] Negotiation successful
0807 06:51:49.880156 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":410,"threads_started":1}
I20260807 06:51:49.885788 14607 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40407
I20260807 06:51:49.886171 14607 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:49.887189 14607 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
W20260807 06:51:49.891353 14607 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40407 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:51:49.891669 14607 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:49.892027 14607 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:49.893957 14607 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:49.894358 14607 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
W20260807 06:51:49.895812 14607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 06:51:50 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786085510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 06:51:50.310165 14611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:50.310781 14611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:50.311030 14611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:50.323064 14611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:50.323341 14611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:50.323596 14611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:51:50.341166 14611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:43743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=36567
--tserver_master_addrs=127.27.54.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:50.343035 14611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:50.344957 14611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:50.369014 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:50.374179 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:50.374406 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:50.375785 14611 server_base.cc:1034] running on GCE node
I20260807 06:51:50.376688 14611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:50.378124 14611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:50.379422 14611 hybrid_clock.cc:648] HybridClock initialized: now 1786085510379355 us; error 52 us; skew 500 ppm
I20260807 06:51:50.379827 14611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:50 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:50.384445 14611 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:51:50.389343 14611 webserver.cc:460] Webserver started at http://127.27.54.66:36567/ using document root <none> and password file <none>
I20260807 06:51:50.390344 14611 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:50.390522 14611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:51:50 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:50.431533 14611 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.013s sys 0.004s
I20260807 06:51:50.451347 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:50.453799 14611 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.009s sys 0.005s
I20260807 06:51:50.454071 14611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "b65312f2f09844a39d04095ad4f5cd1a"
format_stamp: "Formatted at 2026-08-07 06:51:46 on dist-test-slave-3kk6"
server_key: "fc461f01284847fe6a4a4534767bffe7"
server_key_iv: "8533678039334fb0910008b7e8e376b9"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:50.455807 14611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:50.525950 14611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:50.535367 14611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:50.535825 14611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:50.544056 14611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:51:50.555965 14611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:51:50.556200 14611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 06:51:50.556396 14611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:51:50.556537 14611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:50.658322 14611 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:43743
I20260807 06:51:50.658360 14817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:43743 every 8 connection(s)
I20260807 06:51:50.660233 14611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:51:50.665330 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 14611
I20260807 06:51:50.666301 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.54.84:35007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:41329
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:40407
--webserver_port=39003
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:51:50 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:50.702917 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:50.665950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:59497 (local address 127.27.54.126:40407)
0807 06:51:50.679135 (+ 13185us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:50.679146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:50.679187 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:51:50.679282 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:50.679292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:50.679413 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:50.679610 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:50.685345 (+ 5735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:50.686527 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:50.689189 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:50.689862 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:50.689973 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:50.694811 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:50.694820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:50.694825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:50.694871 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:50.698693 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:50.699776 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:50.699782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:50.699786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:50.699945 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:50.700559 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:50.700564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:50.700567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:50.700949 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:50.701078 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:50.702251 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0807 06:51:50.702496 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12968,"thread_start_us":12790,"threads_started":1}
I20260807 06:51:50.705600 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:50.664984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40407 (local address 127.27.54.66:59497)
0807 06:51:50.670992 (+ 6008us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:50.671067 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:50.674115 (+ 3048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:50.683243 (+ 9128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:50.683259 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:50.684046 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:50.685019 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:50.685051 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:50.686847 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:50.686853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:50.688963 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:50.688982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:50.690389 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:50.690396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:50.690741 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:50.691688 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:50.691721 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:50.694436 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:50.698922 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:50.698933 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:50.698954 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:50.699599 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:50.700115 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:50.700120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:50.700124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:50.700418 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:50.701203 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:50.701212 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:50.701686 (+ 474us) client_negotiation.cc:769] Sending connection context
0807 06:51:50.703531 (+ 1845us) client_negotiation.cc:240] Negotiation successful
0807 06:51:50.704137 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5597,"thread_start_us":5312,"threads_started":1}
I20260807 06:51:50.707958 14819 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40407
I20260807 06:51:50.708293 14819 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:50.709268 14819 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:50.711647 14325 ts_manager.cc:194] Registered new tserver with Master: b65312f2f09844a39d04095ad4f5cd1a (127.27.54.66:43743)
I20260807 06:51:50.713874 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:59497
I20260807 06:51:50.895985 14607 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:50.896564 14607 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:50.898115 14325 ts_manager.cc:194] Registered new tserver with Master: 925c1fd750c247988a63f77387a3311d (127.27.54.65:46683)
I20260807 06:51:50.899744 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:44647
Aug 07 06:51:50 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
W20260807 06:51:51.221694 14823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:51.222262 14823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:51.222465 14823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:51.233850 14823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:51.234109 14823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:51.234335 14823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:51:51.249562 14823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.54.84:35007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:41329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=39003
--tserver_master_addrs=127.27.54.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:51.251681 14823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:51.253556 14823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 07 06:51:51 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
W20260807 06:51:51.298730 14831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:51.298544 14832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:51.304204 14834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:51.305163 14823 server_base.cc:1034] running on GCE node
I20260807 06:51:51.306216 14823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:51.313968 14823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:51.315449 14823 hybrid_clock.cc:648] HybridClock initialized: now 1786085511315340 us; error 106 us; skew 500 ppm
I20260807 06:51:51.315929 14823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:51 dist-test-slave-3kk6 krb5kdc[12948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:51.320741 14823 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:51:51.340873 14823 webserver.cc:460] Webserver started at http://127.27.54.67:39003/ using document root <none> and password file <none>
I20260807 06:51:51.342085 14823 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:51.342374 14823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 06:51:51 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for HTTP/127.27.54.84@KRBTEST.COM
I20260807 06:51:51.381248 14823 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.004s
I20260807 06:51:51.396483 14843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:51.401037 14823 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.011s sys 0.000s
I20260807 06:51:51.401393 14823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "36ab11e846184a9990fedaa51fb2f2ef"
format_stamp: "Formatted at 2026-08-07 06:51:47 on dist-test-slave-3kk6"
server_key: "9b4cc843491ba1e15b3d69b018621748"
server_key_iv: "e7c063caf6d7781a981bac5d51e5c5b5"
server_key_version: "kuduclusterkey@0"
I20260807 06:51:51.404081 14823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:51.461073 14823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:51.467584 14823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:51.468097 14823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:51.485101 14823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:51:51.497371 14823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:51:51.497583 14823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 06:51:51.497776 14823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:51:51.497907 14823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:51.584384 14823 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:41329
I20260807 06:51:51.584498 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:41329 every 8 connection(s)
I20260807 06:51:51.585955 14823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:51:51.587145 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 14823
Aug 07 06:51:51 dist-test-slave-3kk6 krb5kdc[12948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085511, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:51.625422 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.590871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.27.54.126:40407)
0807 06:51:51.607281 (+ 16410us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.607290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.607322 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:51:51.607429 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.607438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.607545 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.610548 (+ 3003us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:51.612000 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.613183 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.615331 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.616061 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.616208 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.619208 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:51.619213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:51.619217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:51.619255 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:51.622382 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:51.623185 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:51.623194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:51.623196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:51.623317 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:51.623776 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:51.623781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:51.623784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:51.624115 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:51.624407 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.624796 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.625045 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16242,"thread_start_us":16999,"threads_started":1}
I20260807 06:51:51.631278 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.591256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:50591 (local address 127.27.54.126:40407)
0807 06:51:51.608190 (+ 16934us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.608195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.609433 (+ 1238us) server_negotiation.cc:407] Connection header received
0807 06:51:51.613120 (+ 3687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.613126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.613227 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.613367 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:51.615729 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.617132 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.619551 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.620254 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.620384 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.624552 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:51.624561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:51.624569 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:51.624615 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:51.627831 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:51.628717 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:51.628722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:51.628726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:51.628879 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:51.629414 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:51.629419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:51.629422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:51.629769 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:51.629918 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.630645 (+ 727us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.630906 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16791,"thread_start_us":15738,"threads_started":1}
I20260807 06:51:51.632267 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.589563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40407 (local address 127.27.54.67:50591)
0807 06:51:51.609094 (+ 19531us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:51.609158 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:51.612777 (+ 3619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:51.613941 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:51.613955 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:51.614642 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:51.615497 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.615518 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.617306 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.617313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.619356 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.619373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.620733 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.620741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.621044 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.621761 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:51.621792 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:51.624251 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:51.627975 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:51.627985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:51.628004 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:51.628584 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:51.629029 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:51.629034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:51.629037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:51.629280 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:51.629903 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:51.629909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:51.630254 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 06:51:51.630489 (+ 235us) client_negotiation.cc:240] Negotiation successful
0807 06:51:51.631020 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18984,"spinlock_wait_cycles":268672,"thread_start_us":18844,"threads_started":1}
I20260807 06:51:51.633764 15005 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40407
I20260807 06:51:51.634095 15005 heartbeater.cc:464] Registering TS with master...
I20260807 06:51:51.634977 15005 heartbeater.cc:511] Master 127.27.54.126:40407 requested a full tablet report, sending...
I20260807 06:51:51.636471 14325 ts_manager.cc:194] Registered new tserver with Master: 36ab11e846184a9990fedaa51fb2f2ef (127.27.54.67:41329)
I20260807 06:51:51.637286 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:50591
I20260807 06:51:51.641430 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:51:51.661648 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.645165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.27.54.126:40407)
0807 06:51:51.645389 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.645395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.645915 (+ 520us) server_negotiation.cc:407] Connection header received
0807 06:51:51.646077 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.646083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.646161 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.646267 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:51.647871 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.648936 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.650236 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.650921 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.651048 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.654618 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:51.654625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:51.654630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:51.654704 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:51.658297 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:51.659298 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:51.659304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:51.659308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:51.659464 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:51.659948 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:51.659954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:51.659957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:51.660331 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:51.660584 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.661087 (+ 503us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.661319 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:51:51.668197 14325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:51:51.679039 14325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:51:51.719084 14819 heartbeater.cc:503] Master 127.27.54.126:40407 was elected leader, sending a full tablet report...
I20260807 06:51:51.720824 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.695292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:43743 (local address 127.0.0.1:55874)
0807 06:51:51.696114 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:51.696154 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:51.696338 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:51.710276 (+ 13938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:51.710302 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:51.710353 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:51:51.710910 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.710929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.713946 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.713955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.717065 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.717090 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.719590 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.719600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.719827 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.719895 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 06:51:51.720029 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 06:51:51.720325 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":740,"threads_started":1}
I20260807 06:51:51.723207 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.695892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.27.54.66:43743)
0807 06:51:51.709132 (+ 13240us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.709144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.709183 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:51:51.709327 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.709335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.709657 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.711206 (+ 1549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:51:51.711555 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.713696 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.717418 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.719298 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.722231 (+ 2933us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.722588 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.722700 (+ 112us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.722883 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13042,"thread_start_us":13983,"threads_started":1}
I20260807 06:51:51.727803 14723 tablet_service.cc:1467] Processing CreateTablet for tablet 97f8b725464f4484b2b19c966ce5919c (DEFAULT_TABLE table=test-table [id=f056edd9699f4c648abfb7d682400352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:51:51.729766 14723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f8b725464f4484b2b19c966ce5919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:51.734200 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.696166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:41329 (local address 127.0.0.1:54410)
0807 06:51:51.707996 (+ 11830us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:51.708037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:51.708277 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:51.726894 (+ 18617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:51.726900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:51.726930 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:51:51.727331 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.727343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.729418 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.729423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.731522 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.731538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.733439 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.733446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.733624 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.733661 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:51:51.733729 (+ 68us) client_negotiation.cc:240] Negotiation successful
0807 06:51:51.733869 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11616,"thread_start_us":12212,"threads_started":1}
I20260807 06:51:51.735435 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.696809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.27.54.67:41329)
0807 06:51:51.726241 (+ 29432us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.726249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.726283 (+ 34us) server_negotiation.cc:407] Connection header received
0807 06:51:51.726383 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.726390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.726632 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.726878 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:51:51.727616 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.729248 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.731775 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.733163 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.734547 (+ 1384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.734812 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.734889 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.735098 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29262,"thread_start_us":29106,"threads_started":1}
I20260807 06:51:51.736838 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.715542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:46683 (local address 127.0.0.1:56764)
0807 06:51:51.721146 (+ 5604us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:51.721170 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:51.721313 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:51.727250 (+ 5937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:51.727259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:51.727291 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:51:51.727473 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.727484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.730935 (+ 3451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.730940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.732793 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.732819 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.736073 (+ 3254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.736080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.736277 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.736324 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 06:51:51.736396 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 06:51:51.736551 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5459,"thread_start_us":8651,"threads_started":1}
I20260807 06:51:51.738174 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.709392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.27.54.65:46683)
0807 06:51:51.722981 (+ 13589us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.722995 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.723027 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:51:51.723130 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.723139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.723453 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.723690 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:51:51.729060 (+ 5370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.730741 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.734249 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.735866 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.737104 (+ 1238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.737587 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.737667 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.737852 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13397,"thread_start_us":15111,"threads_started":1}
I20260807 06:51:51.739300 14909 tablet_service.cc:1467] Processing CreateTablet for tablet 97f8b725464f4484b2b19c966ce5919c (DEFAULT_TABLE table=test-table [id=f056edd9699f4c648abfb7d682400352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:51:51.740900 14909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f8b725464f4484b2b19c966ce5919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:51.741984 14516 tablet_service.cc:1467] Processing CreateTablet for tablet 97f8b725464f4484b2b19c966ce5919c (DEFAULT_TABLE table=test-table [id=f056edd9699f4c648abfb7d682400352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:51:51.743407 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f8b725464f4484b2b19c966ce5919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:51.757391 15035 tablet_bootstrap.cc:492] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Bootstrap starting.
I20260807 06:51:51.761312 15037 tablet_bootstrap.cc:492] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Bootstrap starting.
I20260807 06:51:51.761469 15035 tablet_bootstrap.cc:654] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:51.762910 15035 log.cc:824] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:51.766101 15037 tablet_bootstrap.cc:654] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:51.767359 15037 log.cc:824] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:51.771754 15040 tablet_bootstrap.cc:492] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Bootstrap starting.
I20260807 06:51:51.776057 15040 tablet_bootstrap.cc:654] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:51.776916 15035 tablet_bootstrap.cc:492] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: No bootstrap required, opened a new log
I20260807 06:51:51.777225 15035 ts_tablet_manager.cc:1397] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20260807 06:51:51.777601 15037 tablet_bootstrap.cc:492] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: No bootstrap required, opened a new log
I20260807 06:51:51.777595 15040 log.cc:824] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:51.777858 15037 ts_tablet_manager.cc:1397] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260807 06:51:51.787864 15037 raft_consensus.cc:348] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.788385 15037 raft_consensus.cc:374] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:51.788496 15037 raft_consensus.cc:729] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ab11e846184a9990fedaa51fb2f2ef, State: Initialized, Role: FOLLOWER
I20260807 06:51:51.789266 15037 consensus_queue.cc:260] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.791136 15035 raft_consensus.cc:348] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.791688 15035 raft_consensus.cc:374] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:51.791813 15035 raft_consensus.cc:729] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65312f2f09844a39d04095ad4f5cd1a, State: Initialized, Role: FOLLOWER
I20260807 06:51:51.792587 15005 heartbeater.cc:503] Master 127.27.54.126:40407 was elected leader, sending a full tablet report...
I20260807 06:51:51.792616 15035 consensus_queue.cc:260] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.793700 15037 ts_tablet_manager.cc:1428] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260807 06:51:51.794526 15040 tablet_bootstrap.cc:492] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: No bootstrap required, opened a new log
I20260807 06:51:51.794848 15040 ts_tablet_manager.cc:1397] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.010s
I20260807 06:51:51.801719 15035 ts_tablet_manager.cc:1428] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Time spent starting tablet: real 0.024s user 0.016s sys 0.008s
I20260807 06:51:51.804824 15040 raft_consensus.cc:348] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.805297 15040 raft_consensus.cc:374] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:51.805397 15040 raft_consensus.cc:729] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925c1fd750c247988a63f77387a3311d, State: Initialized, Role: FOLLOWER
I20260807 06:51:51.806034 15040 consensus_queue.cc:260] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.807166 14607 heartbeater.cc:503] Master 127.27.54.126:40407 was elected leader, sending a full tablet report...
I20260807 06:51:51.807884 15040 ts_tablet_manager.cc:1428] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260807 06:51:51.851089 14608 tablet.cc:2369] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:51:51.860918 15006 tablet.cc:2369] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:51:51.931712 14820 tablet.cc:2369] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:51:51.974218 15047 raft_consensus.cc:482] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:51:51.974552 15047 raft_consensus.cc:504] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.976490 15047 leader_election.cc:288] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36ab11e846184a9990fedaa51fb2f2ef (127.27.54.67:41329), 925c1fd750c247988a63f77387a3311d (127.27.54.65:46683)
I20260807 06:51:51.986739 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.976957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:56181 (local address 127.27.54.67:41329)
0807 06:51:51.977207 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.977215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.977897 (+ 682us) server_negotiation.cc:407] Connection header received
0807 06:51:51.978189 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.978197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.978298 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.978379 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:51:51.979511 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.980703 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.984048 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.985383 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.985597 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.985686 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.986304 (+ 618us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.986436 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:51:51.986726 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.978172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:49575 (local address 127.27.54.65:46683)
0807 06:51:51.978392 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:51.978396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:51.978553 (+ 157us) server_negotiation.cc:407] Connection header received
0807 06:51:51.978791 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:51.978795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:51.978854 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:51.978935 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:51:51.979798 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.980876 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.984040 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.985383 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.985634 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.985720 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:51.986304 (+ 584us) server_negotiation.cc:299] Negotiation successful
0807 06:51:51.986436 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:51:51.987222 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.977739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:46683 (local address 127.27.54.66:49575)
0807 06:51:51.978452 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:51.978470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:51.978608 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:51.978994 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:51.978999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:51.979021 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:51:51.979620 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.979631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.981036 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.981042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.983870 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.983888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.985787 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.985792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.986073 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.986151 (+ 78us) client_negotiation.cc:769] Sending connection context
0807 06:51:51.986265 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 06:51:51.986483 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":90,"thread_start_us":378,"threads_started":1}
I20260807 06:51:51.987263 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:51.976862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:41329 (local address 127.27.54.66:56181)
0807 06:51:51.977728 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:51.977754 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:51.977945 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:51.978638 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:51.978643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:51.978701 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:51:51.979337 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.979352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.980871 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.980875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.983879 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:51.983888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:51.985747 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:51.985757 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:51.986080 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:51.986159 (+ 79us) client_negotiation.cc:769] Sending connection context
0807 06:51:51.986265 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 06:51:51.986484 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":150,"thread_start_us":484,"threads_started":1}
I20260807 06:51:51.988725 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8b725464f4484b2b19c966ce5919c" candidate_uuid: "b65312f2f09844a39d04095ad4f5cd1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ab11e846184a9990fedaa51fb2f2ef" is_pre_election: true
I20260807 06:51:51.988761 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8b725464f4484b2b19c966ce5919c" candidate_uuid: "b65312f2f09844a39d04095ad4f5cd1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925c1fd750c247988a63f77387a3311d" is_pre_election: true
I20260807 06:51:51.989455 14935 raft_consensus.cc:2454] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65312f2f09844a39d04095ad4f5cd1a in term 0.
I20260807 06:51:51.989455 14543 raft_consensus.cc:2454] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b65312f2f09844a39d04095ad4f5cd1a in term 0.
I20260807 06:51:51.990243 14657 leader_election.cc:302] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 925c1fd750c247988a63f77387a3311d, b65312f2f09844a39d04095ad4f5cd1a; no voters:
I20260807 06:51:51.990635 15047 raft_consensus.cc:2781] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:51:51.990880 15047 raft_consensus.cc:482] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:51:51.991019 15047 raft_consensus.cc:3037] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:51.993420 15047 raft_consensus.cc:504] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:51.994035 15047 leader_election.cc:288] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [CANDIDATE]: Term 1 election: Requested vote from peers 36ab11e846184a9990fedaa51fb2f2ef (127.27.54.67:41329), 925c1fd750c247988a63f77387a3311d (127.27.54.65:46683)
I20260807 06:51:51.994596 14935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8b725464f4484b2b19c966ce5919c" candidate_uuid: "b65312f2f09844a39d04095ad4f5cd1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ab11e846184a9990fedaa51fb2f2ef"
I20260807 06:51:51.994688 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8b725464f4484b2b19c966ce5919c" candidate_uuid: "b65312f2f09844a39d04095ad4f5cd1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925c1fd750c247988a63f77387a3311d"
I20260807 06:51:51.994903 14543 raft_consensus.cc:3037] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:51.994923 14935 raft_consensus.cc:3037] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:51.997372 14543 raft_consensus.cc:2454] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65312f2f09844a39d04095ad4f5cd1a in term 1.
I20260807 06:51:51.997527 14935 raft_consensus.cc:2454] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b65312f2f09844a39d04095ad4f5cd1a in term 1.
I20260807 06:51:51.997856 14657 leader_election.cc:302] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 925c1fd750c247988a63f77387a3311d, b65312f2f09844a39d04095ad4f5cd1a; no voters:
I20260807 06:51:51.998227 15047 raft_consensus.cc:2781] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:51:51.998843 15047 raft_consensus.cc:686] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 1 LEADER]: Becoming Leader. State: Replica: b65312f2f09844a39d04095ad4f5cd1a, State: Running, Role: LEADER
I20260807 06:51:51.999440 15047 consensus_queue.cc:237] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } }
I20260807 06:51:52.004519 14324 catalog_manager.cc:5495] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a reported cstate change: term changed from 0 to 1, leader changed from <none> to b65312f2f09844a39d04095ad4f5cd1a (127.27.54.66). New cstate: current_term: 1 leader_uuid: "b65312f2f09844a39d04095ad4f5cd1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65312f2f09844a39d04095ad4f5cd1a" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 43743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 } health_report { overall_health: UNKNOWN } } }
I20260807 06:51:52.039955 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:52.034428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.27.54.66:43743)
0807 06:51:52.034626 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:52.034631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:52.034691 (+ 60us) server_negotiation.cc:407] Connection header received
0807 06:51:52.034815 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:52.034819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:52.034895 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:52.035018 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:51:52.035391 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.036250 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.037811 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.038252 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.038349 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:52.038946 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:51:52.039461 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:51:52.039569 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:52.039673 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 06:51:52.039743 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:51:52.048985 14935 raft_consensus.cc:1261] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 1 FOLLOWER]: Refusing update from remote peer b65312f2f09844a39d04095ad4f5cd1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:51:52.048987 14543 raft_consensus.cc:1261] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 1 FOLLOWER]: Refusing update from remote peer b65312f2f09844a39d04095ad4f5cd1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:51:52.049983 15047 consensus_queue.cc:1035] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ab11e846184a9990fedaa51fb2f2ef" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:51:52.050334 15052 consensus_queue.cc:1035] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "925c1fd750c247988a63f77387a3311d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:51:52.078763 14325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:51:52.079157 14325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:51:52.083317 14325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f0e5792a35b4bb9b3f2a80f03db37b5: Sending DeleteTablet for 3 replicas of tablet 97f8b725464f4484b2b19c966ce5919c
I20260807 06:51:52.084653 14723 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f8b725464f4484b2b19c966ce5919c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:51:52 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:55874
I20260807 06:51:52.084789 14909 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f8b725464f4484b2b19c966ce5919c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:51:52 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:54410
I20260807 06:51:52.084883 14516 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f8b725464f4484b2b19c966ce5919c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:51:52 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:56764
I20260807 06:51:52.102437 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 14380
I20260807 06:51:52.102532 15072 tablet_replica.cc:330] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: stopping tablet replica
I20260807 06:51:52.103142 15072 raft_consensus.cc:2229] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:51:52.103888 15072 raft_consensus.cc:2258] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:51:52.106364 15072 ts_tablet_manager.cc:1905] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:51:52.114692 15072 ts_tablet_manager.cc:1918] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:51:52.115089 15072 log.cc:1196] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/97f8b725464f4484b2b19c966ce5919c
I20260807 06:51:52.115834 15072 ts_tablet_manager.cc:1939] T 97f8b725464f4484b2b19c966ce5919c P 925c1fd750c247988a63f77387a3311d: Deleting consensus metadata
I20260807 06:51:52.118286 14301 catalog_manager.cc:4841] TS 925c1fd750c247988a63f77387a3311d (127.27.54.65:46683): tablet 97f8b725464f4484b2b19c966ce5919c (table test-table [id=f056edd9699f4c648abfb7d682400352]) successfully deleted
I20260807 06:51:52.127508 15068 tablet_replica.cc:330] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: stopping tablet replica
I20260807 06:51:52.128165 15068 raft_consensus.cc:2229] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:51:52.129175 15068 raft_consensus.cc:2258] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:51:52.131946 15068 ts_tablet_manager.cc:1905] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:51:52.134737 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:52.105868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.27.54.65:46683)
0807 06:51:52.106093 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:52.106101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:52.106128 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:51:52.106321 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:52.106329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:52.106454 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:52.106572 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:52.109592 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.111336 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.114916 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.116006 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.116376 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:52.122349 (+ 5973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:52.122373 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:52.122399 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:52.122462 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:52.128254 (+ 5792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:52.129647 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:52.129660 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:52.129666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:52.129916 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:52.130531 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:52.130539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:52.130545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:52.131199 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:52.133604 (+ 2405us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:52.133821 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 06:51:52.134151 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 06:51:52.135586 14600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:52.135615 15070 tablet_replica.cc:330] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: stopping tablet replica
I20260807 06:51:52.136212 15070 raft_consensus.cc:2229] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:51:52.136873 15070 raft_consensus.cc:2258] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:51:52.139385 15070 ts_tablet_manager.cc:1905] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:51:52.143139 15068 ts_tablet_manager.cc:1918] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:51:52.143424 15068 log.cc:1196] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/97f8b725464f4484b2b19c966ce5919c
I20260807 06:51:52.144225 15068 ts_tablet_manager.cc:1939] T 97f8b725464f4484b2b19c966ce5919c P b65312f2f09844a39d04095ad4f5cd1a: Deleting consensus metadata
I20260807 06:51:52.146759 14302 catalog_manager.cc:4841] TS b65312f2f09844a39d04095ad4f5cd1a (127.27.54.66:43743): tablet 97f8b725464f4484b2b19c966ce5919c (table test-table [id=f056edd9699f4c648abfb7d682400352]) successfully deleted
I20260807 06:51:52.147459 15070 ts_tablet_manager.cc:1918] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:51:52.147737 15070 log.cc:1196] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/97f8b725464f4484b2b19c966ce5919c
I20260807 06:51:52.148589 15070 ts_tablet_manager.cc:1939] T 97f8b725464f4484b2b19c966ce5919c P 36ab11e846184a9990fedaa51fb2f2ef: Deleting consensus metadata
I20260807 06:51:52.150899 14301 catalog_manager.cc:4841] TS 36ab11e846184a9990fedaa51fb2f2ef (127.27.54.67:41329): tablet 97f8b725464f4484b2b19c966ce5919c (table test-table [id=f056edd9699f4c648abfb7d682400352]) successfully deleted
I20260807 06:51:52.239969 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 14380
I20260807 06:51:52.268611 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 14611
I20260807 06:51:52.284427 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:52.269648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.27.54.66:43743)
0807 06:51:52.269856 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:52.269864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:52.269890 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:51:52.269970 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:52.269974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:52.270034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:52.270158 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:52.271663 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.272524 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.273879 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.274398 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.274545 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:52.277452 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:52.277472 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:52.277488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:52.277523 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:52.281113 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:52.281929 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:52.281938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:52.281940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:52.282107 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:52.282521 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:52.282527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:52.282530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:52.283120 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:52.283480 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:52.283808 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:51:52.284061 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:51:52.285300 14813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:52.403716 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 14611
I20260807 06:51:52.429797 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 14823
I20260807 06:51:52.444089 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:52.430699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.27.54.67:41329)
0807 06:51:52.430895 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:52.430901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:52.430925 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:51:52.431070 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:52.431074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:52.431146 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:52.431238 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:52.432663 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.433395 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.434853 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:52.435250 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:52.435352 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:52.438262 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:52.438280 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:52.438296 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:52.438334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:52.441393 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:52.442119 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:52.442128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:52.442130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:52.442283 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:52.442617 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:52.442620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:52.442622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:52.443122 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:52.443349 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:52.443619 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 06:51:52.443818 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:51:52.444830 14999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:52.536448 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 14823
I20260807 06:51:52.563023 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 14276
I20260807 06:51:52.564419 14370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:52.683017 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 14276
I20260807 06:51:52.716181 27865 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 06:51:53.062721 27865 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 06:51:53.063036 27865 mini_ranger.cc:66] Stopping Ranger...
I20260807 06:51:53.420181 27865 mini_ranger.cc:68] Stopped Ranger
2026-08-07 06:51:53.420 UTC [12987] LOG: received smart shutdown request
2026-08-07 06:51:53.423 UTC [12987] LOG: background worker "logical replication launcher" (PID 13001) exited with exit code 1
2026-08-07 06:51:53.423 UTC [12992] LOG: shutting down
2026-08-07 06:51:53.612 UTC [12987] LOG: database system is shut down
2026-08-07T06:51:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58651 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:51:53 dist-test-slave-3kk6 krb5kdc[15091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:51:53 dist-test-slave-3kk6 krb5kdc[15091](info): set up 2 sockets
Aug 07 06:51:53 dist-test-slave-3kk6 krb5kdc[15091](info): commencing operation
krb5kdc: starting...
W20260807 06:51:55.066090 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:51:55 dist-test-slave-3kk6 krb5kdc[15091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:51:55.391410 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:45637
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:45637,127.27.54.125:38057,127.27.54.124:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:51:55.766891 15113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:55.767230 15113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:55.767424 15113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:55.775308 15113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:51:55.775435 15113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:55.775535 15113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:55.775641 15113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:51:55.775736 15113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:51:55.786502 15113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:45637,127.27.54.125:38057,127.27.54.124:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:45637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:55.788069 15113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:55.789556 15113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:55.802551 15119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:55.802534 15120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:55.802593 15122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:55.803849 15113 server_base.cc:1034] running on GCE node
I20260807 06:51:55.804831 15113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:55.806444 15113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:55.807701 15113 hybrid_clock.cc:648] HybridClock initialized: now 1786085515807656 us; error 68 us; skew 500 ppm
I20260807 06:51:55.808106 15113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:55 dist-test-slave-3kk6 krb5kdc[15091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:55.815294 15113 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:51:55.817117 15113 webserver.cc:460] Webserver started at http://127.27.54.126:38631/ using document root <none> and password file <none>
I20260807 06:51:55.817919 15113 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:55.818094 15113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:55.818476 15113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:51:55.821882 15113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "ac349953b16549fe9c9654b60d9c95b9"
format_stamp: "Formatted at 2026-08-07 06:51:55 on dist-test-slave-3kk6"
server_key: "41536e8c4a24f1c787e4e52e02985956"
server_key_iv: "e37cbe0c1e38491cd42216944fdc81d8"
server_key_version: "encryptionkey@0"
I20260807 06:51:55.822813 15113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "ac349953b16549fe9c9654b60d9c95b9"
format_stamp: "Formatted at 2026-08-07 06:51:55 on dist-test-slave-3kk6"
server_key: "41536e8c4a24f1c787e4e52e02985956"
server_key_iv: "e37cbe0c1e38491cd42216944fdc81d8"
server_key_version: "encryptionkey@0"
I20260807 06:51:55.828346 15113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 06:51:55.832113 15131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:55.833631 15113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:51:55.833864 15113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "ac349953b16549fe9c9654b60d9c95b9"
format_stamp: "Formatted at 2026-08-07 06:51:55 on dist-test-slave-3kk6"
server_key: "41536e8c4a24f1c787e4e52e02985956"
server_key_iv: "e37cbe0c1e38491cd42216944fdc81d8"
server_key_version: "encryptionkey@0"
I20260807 06:51:55.834098 15113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:55.869757 15113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:55.874260 15113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:55.874605 15113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:55.894960 15113 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:45637
I20260807 06:51:55.894995 15196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:45637 every 8 connection(s)
I20260807 06:51:55.896565 15113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:51:55.900549 15197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:55.906307 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15113
I20260807 06:51:55.906559 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:51:55.907258 27865 external_mini_cluster.cc:1428] Setting key 6b7944a6600edbedadcecf0428b2737c
I20260807 06:51:55.907626 15197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:51:55 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:55.931854 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:55.911212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.27.54.126:45637)
0807 06:51:55.912143 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:55.912166 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:55.912705 (+ 539us) server_negotiation.cc:407] Connection header received
0807 06:51:55.914919 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:55.914925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:55.915172 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:55.915390 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:55.916740 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:55.918002 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:55.919600 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:55.920177 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:55.920389 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:55.924135 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:55.924144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:55.924157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:55.924189 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:55.927225 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:55.928011 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:55.928018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:55.928020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:55.928168 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:55.928585 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:55.928596 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:55.928598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:55.929123 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:55.929357 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:55.929904 (+ 547us) server_negotiation.cc:299] Negotiation successful
0807 06:51:55.930288 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":679,"thread_start_us":439,"threads_started":1}
I20260807 06:51:55.931893 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:55.911230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:45637 (local address 127.0.0.1:47312)
0807 06:51:55.912513 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:55.912551 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:55.914460 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:55.915416 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:55.915422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:55.916167 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:55.916528 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:55.916545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:55.918179 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:55.918189 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:55.919448 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:55.919461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:55.920433 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:55.920439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:55.920612 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:55.921385 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:55.921408 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:55.923875 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:55.927400 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:55.927413 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:55.927426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:55.927859 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:55.928298 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:55.928303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:55.928306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:55.928466 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:55.929273 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:55.929282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:55.929515 (+ 233us) client_negotiation.cc:769] Sending connection context
0807 06:51:55.929790 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 06:51:55.930406 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":163,"spinlock_wait_cycles":17152,"thread_start_us":814,"threads_started":1}
I20260807 06:51:55.935297 15197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.54.125@KRBTEST.COM; defaulting to no policy
I20260807 06:51:55.936607 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:55.936175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42194)
0807 06:51:55.936346 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:55.936489 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 06:51:55.936942 15135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
I20260807 06:51:55.938706 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:55.938199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42210)
0807 06:51:55.938385 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:55.938459 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260807 06:51:55.938971 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38057: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab.
Entry for principal kudu/127.27.54.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab.
I20260807 06:51:55.975145 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:38057
--webserver_interface=127.27.54.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab
--principal=kudu/127.27.54.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:45637,127.27.54.125:38057,127.27.54.124:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:51:55.988499 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } attempt: 1
I20260807 06:51:55.989868 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:55.989370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42226)
0807 06:51:55.989579 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:55.989714 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 06:51:55.991106 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:55.990724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42234)
0807 06:51:55.990879 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:55.990967 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 06:51:55.991339 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38057: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
I20260807 06:51:56.059692 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } attempt: 2
I20260807 06:51:56.061074 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.060559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42248)
0807 06:51:56.060836 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.060940 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 06:51:56.062109 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.061770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42250)
0807 06:51:56.061921 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.061988 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 06:51:56.062331 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38057: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
I20260807 06:51:56.153581 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } attempt: 3
I20260807 06:51:56.154980 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.154460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42266)
0807 06:51:56.154722 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.154839 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 06:51:56.156085 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.155693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42274)
0807 06:51:56.155862 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.155948 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 06:51:56.156278 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38057: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
I20260807 06:51:56.299561 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } attempt: 4
I20260807 06:51:56.300863 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.300260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42282)
0807 06:51:56.300616 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.300718 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260807 06:51:56.301932 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.301590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42294)
0807 06:51:56.301735 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.301816 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 06:51:56.302138 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38057: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38057: connect: Connection refused (error 111)
W20260807 06:51:56.331380 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:56.331785 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:56.331964 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:56.339653 15207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:51:56.339810 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:56.339922 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:56.340023 15207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:51:56.340109 15207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:51:56.350245 15207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:45637,127.27.54.125:38057,127.27.54.124:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab
--principal=kudu/127.27.54.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.54.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:56.351809 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:56.353226 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:56.365562 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:56.365617 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:56.366122 15218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:56.366628 15207 server_base.cc:1034] running on GCE node
I20260807 06:51:56.367558 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:56.369019 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:56.370255 15207 hybrid_clock.cc:648] HybridClock initialized: now 1786085516370217 us; error 50 us; skew 500 ppm
I20260807 06:51:56.370610 15207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:56 dist-test-slave-3kk6 krb5kdc[15091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:56.373833 15207 init.cc:377] Logged in from keytab as kudu/127.27.54.125@KRBTEST.COM (short username kudu)
I20260807 06:51:56.375373 15207 webserver.cc:460] Webserver started at http://127.27.54.125:46217/ using document root <none> and password file <none>
I20260807 06:51:56.376094 15207 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:56.376262 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:56.376581 15207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:51:56.379700 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data/instance:
uuid: "df291e685be346a2ae3a2e896d836b56"
format_stamp: "Formatted at 2026-08-07 06:51:56 on dist-test-slave-3kk6"
server_key: "4841c31b2fca87e891224293f10d22c3"
server_key_iv: "cddfc53f451e370a734fc2629c275864"
server_key_version: "encryptionkey@0"
I20260807 06:51:56.380538 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal/instance:
uuid: "df291e685be346a2ae3a2e896d836b56"
format_stamp: "Formatted at 2026-08-07 06:51:56 on dist-test-slave-3kk6"
server_key: "4841c31b2fca87e891224293f10d22c3"
server_key_iv: "cddfc53f451e370a734fc2629c275864"
server_key_version: "encryptionkey@0"
I20260807 06:51:56.385942 15207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 06:51:56.389679 15229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:56.391247 15207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:51:56.391449 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal
uuid: "df291e685be346a2ae3a2e896d836b56"
format_stamp: "Formatted at 2026-08-07 06:51:56 on dist-test-slave-3kk6"
server_key: "4841c31b2fca87e891224293f10d22c3"
server_key_iv: "cddfc53f451e370a734fc2629c275864"
server_key_version: "encryptionkey@0"
I20260807 06:51:56.391733 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:56.404897 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:56.409288 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:56.409607 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:56.429389 15207 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.125:38057
I20260807 06:51:56.429427 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.125:38057 every 8 connection(s)
I20260807 06:51:56.430968 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
I20260807 06:51:56.434983 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:56.438917 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15207
I20260807 06:51:56.439179 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-1/wal/instance
I20260807 06:51:56.439647 27865 external_mini_cluster.cc:1428] Setting key 626be93105e0adc2bb0868b9db2708e9
I20260807 06:51:56.442000 15292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:51:56 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:56.463477 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.444794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.27.54.126:45637)
0807 06:51:56.445291 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:56.445296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:56.446498 (+ 1202us) server_negotiation.cc:407] Connection header received
0807 06:51:56.448585 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:56.448591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:56.448686 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:56.448822 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:56.450384 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.451201 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.452823 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.453230 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.453370 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:56.457204 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:56.457210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:56.457213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:56.457262 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:56.460148 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:56.461012 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:56.461018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:56.461020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:56.461145 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:56.461594 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:56.461600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:56.461603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:56.461900 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:56.462072 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:56.462841 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 06:51:56.463070 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":282,"threads_started":1}
I20260807 06:51:56.464599 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.445484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:45637 (local address 127.0.0.1:47314)
0807 06:51:56.446326 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.446366 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:56.448291 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:56.449053 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:56.449061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:56.449607 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:56.450184 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:56.450201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.451381 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.451385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:56.452651 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:56.452664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.453477 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.453485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:56.453718 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:56.454501 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:56.454524 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:56.456925 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:56.460316 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:56.460327 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:56.460346 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:56.460863 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:56.461264 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:56.461269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:56.461272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:56.461473 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:56.462045 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:56.462053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:56.462430 (+ 377us) client_negotiation.cc:769] Sending connection context
0807 06:51:56.462738 (+ 308us) client_negotiation.cc:240] Negotiation successful
0807 06:51:56.463293 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":365,"threads_started":1}
I20260807 06:51:56.466763 15292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.54.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.124@KRBTEST.COM" created.
Aug 07 06:51:56 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:51:56.484258 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.467744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42304)
0807 06:51:56.467975 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.467994 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:56.468112 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:56.469247 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:56.469251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:56.469625 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:56.469784 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:56.469793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.471334 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.471338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:56.472452 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:56.472468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.473384 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.473393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:56.473657 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:56.474336 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:56.474365 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:56.477054 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:56.481626 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:56.481631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:56.481633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:56.482029 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:56.482454 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:56.482457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:56.482459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:56.482610 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:56.483497 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:56.483501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:56.483622 (+ 121us) client_negotiation.cc:769] Sending connection context
0807 06:51:56.483761 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 06:51:56.483985 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 06:51:56.484752 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.467869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.27.54.125:38057)
0807 06:51:56.468694 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:56.468702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:56.468736 (+ 34us) server_negotiation.cc:407] Connection header received
0807 06:51:56.468801 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:56.468809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:56.469074 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:56.469275 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:56.469934 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.471178 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.472645 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.473213 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.473315 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:56.477294 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:56.477313 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:56.477335 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:56.477377 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:56.481461 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:56.482167 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:56.482179 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:56.482185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:56.482338 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:56.482842 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:56.482849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:56.482855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:56.483366 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:56.483588 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:56.483963 (+ 375us) server_negotiation.cc:299] Negotiation successful
0807 06:51:56.484270 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":458,"threads_started":1}
I20260807 06:51:56.487287 15292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:51:56.488669 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.488056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58682)
0807 06:51:56.488285 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.488438 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 06:51:56.488984 15232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.490877 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.490360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58692)
0807 06:51:56.490559 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.490716 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 06:51:56.491194 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab.
Entry for principal kudu/127.27.54.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab.
I20260807 06:51:56.511121 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:36017
--webserver_interface=127.27.54.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab
--principal=kudu/127.27.54.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:45637,127.27.54.125:38057,127.27.54.124:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:51:56.540572 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 1
I20260807 06:51:56.541980 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.541459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58708)
0807 06:51:56.541698 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.541819 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 06:51:56.543191 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.542822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58722)
0807 06:51:56.542997 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.543068 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 06:51:56.543397 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.601418 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } attempt: 5
Aug 07 06:51:56 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:51:56.611614 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 2
I20260807 06:51:56.612777 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.612287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58738)
0807 06:51:56.612498 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.612599 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 06:51:56.614221 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.613726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58742)
0807 06:51:56.613923 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.614024 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 06:51:56.614542 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.617834 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.602306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42306)
0807 06:51:56.602568 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.602588 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:56.602801 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:56.603259 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:56.603262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:56.603661 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:56.603883 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:56.603893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.605329 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.605334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:56.606253 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:56.606267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.607307 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.607311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:56.607447 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:56.608179 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:56.608200 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:56.610333 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:56.615078 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:56.615085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:56.615090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:56.615531 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:56.615960 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:56.615965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:56.615970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:56.616122 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:56.616718 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:56.616724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:56.616920 (+ 196us) client_negotiation.cc:769] Sending connection context
0807 06:51:56.617115 (+ 195us) client_negotiation.cc:240] Negotiation successful
0807 06:51:56.617449 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 06:51:56.617826 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.602412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.27.54.125:38057)
0807 06:51:56.602697 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:56.602704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:56.602731 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:51:56.602958 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:56.602964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:56.603078 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:56.603206 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:56.604073 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.605170 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.606417 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:56.607123 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:56.607264 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:56.610554 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:56.610561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:56.610566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:56.610609 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:56.614911 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:56.615682 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:56.615689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:56.615693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:56.615830 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:56.616266 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:56.616276 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:56.616279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:56.616576 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:56.616758 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:56.617241 (+ 483us) server_negotiation.cc:299] Negotiation successful
0807 06:51:56.617473 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 06:51:56.619194 15197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:51:56.620604 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.620023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58746)
0807 06:51:56.620371 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.620454 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260807 06:51:56.621677 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.621343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58754)
0807 06:51:56.621489 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.621578 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 06:51:56.621912 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.673164 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 1
I20260807 06:51:56.674549 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.673928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58770)
0807 06:51:56.674199 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.674329 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 06:51:56.675743 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.675345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58772)
0807 06:51:56.675509 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.675609 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:51:56.676031 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.705862 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 3
I20260807 06:51:56.707294 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.706774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58778)
0807 06:51:56.707020 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.707124 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 06:51:56.708395 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.707963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58790)
0807 06:51:56.708196 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.708279 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260807 06:51:56.708593 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.744272 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 2
I20260807 06:51:56.745667 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.744953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58804)
0807 06:51:56.745264 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.745408 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 06:51:56.746836 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.746424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58810)
0807 06:51:56.746584 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.746703 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 06:51:56.747151 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.851819 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 4
I20260807 06:51:56.853201 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.852573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58818)
0807 06:51:56.852854 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.852971 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 06:51:56.853441 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 3
I20260807 06:51:56.854457 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.854029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58828)
0807 06:51:56.854215 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.854306 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 06:51:56.854511 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.854065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58830)
0807 06:51:56.854295 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.854383 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260807 06:51:56.854784 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
I20260807 06:51:56.855758 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:56.855276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58838)
0807 06:51:56.855512 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:56.855596 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 06:51:56.856012 15197 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:36017: Network error: Client connection negotiation failed: client connection to 127.27.54.124:36017: connect: Connection refused (error 111)
W20260807 06:51:56.875118 15304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:51:56.875414 15304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:51:56.875591 15304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:51:56.882922 15304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:51:56.883036 15304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:51:56.883121 15304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:51:56.883235 15304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:51:56.883320 15304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:51:56.893738 15304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:45637,127.27.54.125:38057,127.27.54.124:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab
--principal=kudu/127.27.54.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.54.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:51:56.895262 15304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:51:56.896735 15304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:51:56.908917 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:56.908983 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:51:56.908959 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:51:56.909999 15304 server_base.cc:1034] running on GCE node
I20260807 06:51:56.910895 15304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:51:56.912405 15304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:51:56.913604 15304 hybrid_clock.cc:648] HybridClock initialized: now 1786085516913567 us; error 33 us; skew 500 ppm
I20260807 06:51:56.913992 15304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:51:56 dist-test-slave-3kk6 krb5kdc[15091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:51:56.917277 15304 init.cc:377] Logged in from keytab as kudu/127.27.54.124@KRBTEST.COM (short username kudu)
I20260807 06:51:56.918948 15304 webserver.cc:460] Webserver started at http://127.27.54.124:36525/ using document root <none> and password file <none>
I20260807 06:51:56.919662 15304 fs_manager.cc:359] Metadata directory not provided
I20260807 06:51:56.919821 15304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:51:56.920147 15304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:51:56.923362 15304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data/instance:
uuid: "8c5373073ce84b1d9f7ecd3b2a880e37"
format_stamp: "Formatted at 2026-08-07 06:51:56 on dist-test-slave-3kk6"
server_key: "99c497e56caf3ad17de17df1d7b5845d"
server_key_iv: "f992a01dba519582198a4a4f1b11ad72"
server_key_version: "encryptionkey@0"
I20260807 06:51:56.924218 15304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal/instance:
uuid: "8c5373073ce84b1d9f7ecd3b2a880e37"
format_stamp: "Formatted at 2026-08-07 06:51:56 on dist-test-slave-3kk6"
server_key: "99c497e56caf3ad17de17df1d7b5845d"
server_key_iv: "f992a01dba519582198a4a4f1b11ad72"
server_key_version: "encryptionkey@0"
I20260807 06:51:56.929606 15304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 06:51:56.933198 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:51:56.934710 15304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:51:56.934914 15304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal
uuid: "8c5373073ce84b1d9f7ecd3b2a880e37"
format_stamp: "Formatted at 2026-08-07 06:51:56 on dist-test-slave-3kk6"
server_key: "99c497e56caf3ad17de17df1d7b5845d"
server_key_iv: "f992a01dba519582198a4a4f1b11ad72"
server_key_version: "encryptionkey@0"
I20260807 06:51:56.935112 15304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:51:56.975093 15304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:51:56.979666 15304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:51:56.979928 15304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:51:56.999163 15304 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.124:36017
I20260807 06:51:56.999198 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.124:36017 every 8 connection(s)
I20260807 06:51:57.000602 15304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
I20260807 06:51:57.004485 15409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:51:57.004875 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15304
I20260807 06:51:57.005079 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786085287136147-27865-0/minicluster-data/master-2/wal/instance
I20260807 06:51:57.005555 27865 external_mini_cluster.cc:1428] Setting key b3eebdcf468510fb57cb57dbfd9fae77
I20260807 06:51:57.013911 15409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:57.031229 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.007508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.27.54.124:36017)
0807 06:51:57.008350 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.008365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.008411 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:51:57.010597 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.010619 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.011078 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.011421 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.013434 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.014871 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.016760 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.017989 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.018185 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.022868 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.022876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.022888 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.022927 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.026421 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.027325 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.027336 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.027347 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.027550 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.027972 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.027975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.027977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.028734 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.029055 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.029441 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.029881 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":424,"threads_started":1}
I20260807 06:51:57.032483 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.016841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:45637 (local address 127.0.0.1:47316)
0807 06:51:57.017755 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:57.017791 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:57.017990 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:57.018384 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:57.018390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:57.019076 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:57.019343 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.019352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.020342 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.020348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.021685 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.021704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.022527 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.022535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.022790 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.023539 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:57.023556 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:57.025763 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:57.029787 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.029796 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.029801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.030224 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.030730 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.030733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.030736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.030884 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.031568 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:57.031575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:57.031794 (+ 219us) client_negotiation.cc:769] Sending connection context
0807 06:51:57.031957 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 06:51:57.032176 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":501,"threads_started":1}
I20260807 06:51:57.032733 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.016821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.27.54.126:45637)
0807 06:51:57.017385 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.017393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.017899 (+ 506us) server_negotiation.cc:407] Connection header received
0807 06:51:57.018118 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.018129 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.018193 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.018306 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.019498 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.020182 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.021849 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.022263 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.022372 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.025978 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.025985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.025989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.026034 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.029618 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.030354 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.030359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.030363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.030523 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.031009 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.031014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.031017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.031399 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.031551 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.032110 (+ 559us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.032332 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":256,"threads_started":1}
I20260807 06:51:57.033241 15197 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 4
I20260807 06:51:57.034726 15409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:51:57.049779 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.033790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58866)
0807 06:51:57.033979 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:57.034001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:57.034102 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:57.034444 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:57.034449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:57.034949 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:57.035162 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.035177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.036656 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.036663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.037895 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.037908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.039058 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.039065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.039245 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.039933 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:57.039961 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:57.042610 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:57.046155 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.046163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.046169 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.046631 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.047130 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.047138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.047143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.047623 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.048564 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:57.048570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:57.048789 (+ 219us) client_negotiation.cc:769] Sending connection context
0807 06:51:57.048950 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 06:51:57.049281 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 06:51:57.049830 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.033974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.27.54.124:36017)
0807 06:51:57.034123 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.034129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.034158 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:51:57.034206 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.034211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.034286 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.034381 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.035336 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.036505 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.038035 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.038867 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.038999 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.042861 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.042867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.042876 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.042905 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.046024 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.046847 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.046852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.046856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.046991 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.047774 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.047781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.047785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.048155 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.048323 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.049214 (+ 891us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.049455 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:51:57.051532 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.035428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38057 (local address 127.0.0.1:42320)
0807 06:51:57.035819 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:57.035838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:57.035929 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:57.036297 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:57.036302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:57.036794 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:57.036967 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.036978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.038351 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.038356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.039814 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.039833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.040674 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.040679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.040923 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.041523 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:57.041573 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:57.044386 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:57.048396 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.048404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.048410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.048969 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.049460 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.049465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.049469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.049658 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.050474 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:57.050479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:57.050644 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 06:51:57.050861 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 06:51:57.051144 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260807 06:51:57.051766 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.035813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.27.54.125:38057)
0807 06:51:57.036014 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.036019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.036037 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:51:57.036098 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.036102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.036158 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.036270 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.037122 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.038182 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.040039 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.040520 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.040665 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.044612 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.044622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.044628 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.044676 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.048198 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.049108 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.049113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.049116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.049293 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.049860 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.049865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.049868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.050212 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.050365 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.050984 (+ 619us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.051374 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:51:57.052724 15409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:51:57.054059 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.034937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.27.54.126:45637)
0807 06:51:57.035107 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.035133 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.035151 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:51:57.035209 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.035216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.035336 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.035437 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.036998 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.038160 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.040064 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.040776 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.040905 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.046541 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.046549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.046556 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.046598 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.050060 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.050969 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.050974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.050977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.051115 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.051605 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.051613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.051616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.052021 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.052326 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.053492 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.053703 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":58240}
I20260807 06:51:57.056479 15197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: Bootstrap starting.
I20260807 06:51:57.060561 15197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:57.061904 15197 log.cc:824] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:51:57.065528 15197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: No bootstrap required, opened a new log
I20260807 06:51:57.069185 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.053581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58868)
0807 06:51:57.053767 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:57.053787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:57.053890 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:57.054358 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:57.054365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:57.054913 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:57.055102 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.055113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.056866 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.056871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.058130 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.058162 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.059161 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.059169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.059414 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.060102 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:57.060128 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:57.063347 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:57.066407 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.066412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.066416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.066868 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.067279 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.067285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.067290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.067464 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.067999 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:57.068004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:57.068162 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 06:51:57.068323 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 06:51:57.068715 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 06:51:57.069307 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.053891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.27.54.124:36017)
0807 06:51:57.054063 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.054068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.054086 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:51:57.054130 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.054134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.054210 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.054305 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.055255 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.056701 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.058306 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.058995 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.059113 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.063554 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.063560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.063565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.063602 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.066271 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.066984 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.066991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.066994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.067130 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.067577 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.067580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.067582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.067876 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.068047 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.068482 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.068902 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":46}
I20260807 06:51:57.074002 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: Bootstrap starting.
I20260807 06:51:57.074244 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.056112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.27.54.125:38057)
0807 06:51:57.056268 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.056284 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.056361 (+ 77us) server_negotiation.cc:407] Connection header received
0807 06:51:57.056616 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.056624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.056699 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.056804 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.058041 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.059151 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.060608 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.061076 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.061216 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.066502 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.066508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.066513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.066551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.070196 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.071076 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.071082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.071085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.071251 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.071750 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.071756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.071760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.072101 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.072395 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.073634 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.073877 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:51:57.076400 15197 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.076764 15197 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:57.076941 15197 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac349953b16549fe9c9654b60d9c95b9, State: Initialized, Role: FOLLOWER
I20260807 06:51:57.077628 15197 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.078701 15409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:57.079068 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.080060 15197 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:51:57.080098 15409 log.cc:824] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:57.080168 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:51:57.083603 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: No bootstrap required, opened a new log
I20260807 06:51:57.087800 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:51:57.099864 15416 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:51:57.100241 15416 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.103401 15409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.103946 15409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:57.104079 15409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5373073ce84b1d9f7ecd3b2a880e37, State: Initialized, Role: FOLLOWER
I20260807 06:51:57.104758 15409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.104920 15416 leader_election.cc:288] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df291e685be346a2ae3a2e896d836b56 (127.27.54.125:38057), 8c5373073ce84b1d9f7ecd3b2a880e37 (127.27.54.124:36017)
I20260807 06:51:57.108845 15436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.110786 15436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 06:51:57.127707 15138 tablet.cc:2369] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:51:57.130026 15409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:51:57.130363 15365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac349953b16549fe9c9654b60d9c95b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" is_pre_election: true
I20260807 06:51:57.131162 15365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac349953b16549fe9c9654b60d9c95b9 in term 0.
I20260807 06:51:57.131918 15134 leader_election.cc:302] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5373073ce84b1d9f7ecd3b2a880e37, ac349953b16549fe9c9654b60d9c95b9; no voters:
I20260807 06:51:57.132530 15416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:51:57.132731 15416 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:51:57.132870 15416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:57.120594 15113 master_runner.cc:186] Error getting master registration for 127.27.54.125:38057: OK, instance_id { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" instance_seqno: 1786085516423843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:51:57.133904 15113 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:51:57.136471 15416 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.136991 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.086900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.27.54.126:45637)
0807 06:51:57.087098 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.087107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.099738 (+ 12631us) server_negotiation.cc:407] Connection header received
0807 06:51:57.100705 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.100711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.100864 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.100980 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.112953 (+ 11973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.114215 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.116795 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.117482 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.117617 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.126697 (+ 9080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.126706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.126711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.126758 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.132411 (+ 5653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.133863 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.133870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.133875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.134136 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.134732 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.134738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.134741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.135061 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.135387 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.135882 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.136299 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 06:51:57.137300 15416 leader_election.cc:288] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [CANDIDATE]: Term 1 election: Requested vote from peers df291e685be346a2ae3a2e896d836b56 (127.27.54.125:38057), 8c5373073ce84b1d9f7ecd3b2a880e37 (127.27.54.124:36017)
I20260807 06:51:57.138192 15365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac349953b16549fe9c9654b60d9c95b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37"
I20260807 06:51:57.138423 15365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:57.143168 15365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac349953b16549fe9c9654b60d9c95b9 in term 1.
I20260807 06:51:57.143761 15134 leader_election.cc:302] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5373073ce84b1d9f7ecd3b2a880e37, ac349953b16549fe9c9654b60d9c95b9; no voters:
I20260807 06:51:57.144114 15416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 1 FOLLOWER]: Leader election won for term 1
W20260807 06:51:57.144384 15441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:51:57.144554 15441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:51:57.148308 15416 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [term 1 LEADER]: Becoming Leader. State: Replica: ac349953b16549fe9c9654b60d9c95b9, State: Running, Role: LEADER
I20260807 06:51:57.148994 15416 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.151074 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.086385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.27.54.124:36017)
0807 06:51:57.086580 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.086589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.108529 (+ 21940us) server_negotiation.cc:407] Connection header received
0807 06:51:57.108896 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.108905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.109017 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.109129 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.112538 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.113971 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.118231 (+ 4260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.118961 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.119088 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.138727 (+ 19639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.138735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.138739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.138782 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.145234 (+ 6452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.148428 (+ 3194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.148434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.148437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.148598 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.149137 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.149142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.149145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.149546 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.149830 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.150442 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.150734 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:51:57.152091 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.088338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.27.54.125:38057)
0807 06:51:57.088508 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.088516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.111614 (+ 23098us) server_negotiation.cc:407] Connection header received
0807 06:51:57.112605 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.112612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.112712 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.112794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.120514 (+ 7720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.121805 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.133991 (+ 12186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.134738 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.134859 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.141239 (+ 6380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.141247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.141252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.141331 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.147340 (+ 6009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.149440 (+ 2100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.149447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.149450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.149633 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.150113 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.150119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.150123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.150538 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.150800 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.151411 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.151697 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:51:57.153927 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac349953b16549fe9c9654b60d9c95b9. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.154327 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:57.156435 15448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:51:57.157384 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } attempt: 5
I20260807 06:51:57.160606 15448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 06:51:57 dist-test-slave-3kk6 krb5kdc[15091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085516, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
I20260807 06:51:57.177112 15365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [term 1 FOLLOWER]: Refusing update from remote peer ac349953b16549fe9c9654b60d9c95b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:51:57.178349 15446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:51:57.181715 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.158224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.27.54.124:36017)
0807 06:51:57.158397 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 06:51:57.158404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:51:57.158464 (+ 60us) server_negotiation.cc:407] Connection header received
0807 06:51:57.158683 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:51:57.158688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:51:57.158770 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:51:57.158878 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:51:57.160005 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.161114 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.163041 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.163744 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.163878 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.171750 (+ 7872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:51:57.171756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:51:57.171761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:51:57.171798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:51:57.175423 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.178539 (+ 3116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.178545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.178549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.178735 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:51:57.179477 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:51:57.179483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:51:57.179486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:51:57.179885 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:51:57.180029 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 06:51:57.181100 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0807 06:51:57.181340 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:51:57.183512 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:51:57.158152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:36017 (local address 127.0.0.1:58888)
0807 06:51:57.158368 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 06:51:57.158392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:51:57.158541 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:51:57.159078 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:51:57.159082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:51:57.159616 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:51:57.159828 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.159841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.161307 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.161313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.162834 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:51:57.162851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:51:57.164100 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:51:57.164108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:51:57.164372 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:51:57.168411 (+ 4039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:51:57.168439 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:51:57.171534 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:51:57.177814 (+ 6280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.177822 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.177829 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.178406 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.178900 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:51:57.178905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:51:57.178908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:51:57.179248 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:51:57.180218 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:51:57.180222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:51:57.180544 (+ 322us) client_negotiation.cc:769] Sending connection context
0807 06:51:57.180931 (+ 387us) client_negotiation.cc:240] Negotiation successful
0807 06:51:57.182850 (+ 1919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 06:51:57.193629 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: Bootstrap starting.
I20260807 06:51:57.199219 15292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: Neither blocks nor log segments found. Creating new log.
I20260807 06:51:57.200824 15292 log.cc:824] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: Log is configured to *not* fsync() on all Append() calls
I20260807 06:51:57.208240 15436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac349953b16549fe9c9654b60d9c95b9. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.208593 15436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:51:57.213953 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.214236 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:57.215950 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: No bootstrap required, opened a new log
I20260807 06:51:57.217797 15448 catalog_manager.cc:1347] Generated new cluster ID: 84b7cbc0674348d4a0b7ba81fb0060df
I20260807 06:51:57.217896 15448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:51:57.231022 15455 mvcc.cc:205] Tried to move back new op lower bound from 7315806278338273280 to 7315806278249623552. Current Snapshot: MvccSnapshot[applied={T|T < 7315806278338273280}]
I20260807 06:51:57.232252 15292 raft_consensus.cc:348] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.232821 15292 raft_consensus.cc:374] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:51:57.232995 15292 raft_consensus.cc:729] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df291e685be346a2ae3a2e896d836b56, State: Initialized, Role: FOLLOWER
I20260807 06:51:57.233830 15436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.234117 15436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:51:57.234597 15292 consensus_queue.cc:260] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } }
I20260807 06:51:57.236222 15467 sys_catalog.cc:455] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.237063 15467 sys_catalog.cc:458] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:51:57.237617 15260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac349953b16549fe9c9654b60d9c95b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df291e685be346a2ae3a2e896d836b56"
I20260807 06:51:57.237591 15261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac349953b16549fe9c9654b60d9c95b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df291e685be346a2ae3a2e896d836b56" is_pre_election: true
I20260807 06:51:57.238389 15259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:51:57.242734 15259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [term 1 FOLLOWER]: Refusing update from remote peer ac349953b16549fe9c9654b60d9c95b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:51:57.243619 15416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:51:57.236450 15304 master_runner.cc:186] Error getting master registration for 127.27.54.125:38057: OK, instance_id { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" instance_seqno: 1786085516423843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:51:57.245781 15304 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:51:57.237804 15292 sys_catalog.cc:564] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:51:57.264153 15464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:51:57.264313 15464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:51:57.265906 15475 sys_catalog.cc:455] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.266348 15475 sys_catalog.cc:458] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:51:57.266671 15467 sys_catalog.cc:455] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac349953b16549fe9c9654b60d9c95b9. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.266943 15467 sys_catalog.cc:458] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:51:57.267700 15494 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:51:57.279376 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac349953b16549fe9c9654b60d9c95b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac349953b16549fe9c9654b60d9c95b9" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 45637 } } peers { permanent_uuid: "df291e685be346a2ae3a2e896d836b56" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38057 } } peers { permanent_uuid: "8c5373073ce84b1d9f7ecd3b2a880e37" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 36017 } } }
I20260807 06:51:57.279747 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9 [sys.catalog]: This master's current role is: LEADER
I20260807 06:51:57.287387 15448 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:51:57.289350 15448 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:51:57.305531 15448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac349953b16549fe9c9654b60d9c95b9: Generated new TSK 0
I20260807 06:51:57.306762 15448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:51:58.266554 15464 catalog_manager.cc:1259] Loaded cluster ID: 84b7cbc0674348d4a0b7ba81fb0060df
I20260807 06:51:58.266793 15464 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: loading cluster ID for follower catalog manager: success
I20260807 06:51:58.269495 15464 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: acquiring CA information for follower catalog manager: success
I20260807 06:51:58.271849 15464 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c5373073ce84b1d9f7ecd3b2a880e37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:51:58.272992 15494 catalog_manager.cc:1259] Loaded cluster ID: 84b7cbc0674348d4a0b7ba81fb0060df
I20260807 06:51:58.273123 15494 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: loading cluster ID for follower catalog manager: success
I20260807 06:51:58.277140 15494 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: acquiring CA information for follower catalog manager: success
I20260807 06:51:58.279999 15494 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df291e685be346a2ae3a2e896d836b56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:51:58.398476 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15113
I20260807 06:51:58.399776 15192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:58.523113 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15113
W20260807 06:51:58.555534 15335 connection.cc:535] server connection from 127.0.0.1:58866 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 06:51:58.555526 15232 connection.cc:535] server connection from 127.0.0.1:42306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:51:58.556517 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15207
I20260807 06:51:58.557931 15286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:58.675915 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15207
I20260807 06:51:58.703797 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15304
I20260807 06:51:58.705014 15400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:51:58.815732 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15304
2026-08-07T06:51:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5073 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:51:58 dist-test-slave-3kk6 krb5kdc[15514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:51:58 dist-test-slave-3kk6 krb5kdc[15514](info): set up 2 sockets
Aug 07 06:51:58 dist-test-slave-3kk6 krb5kdc[15514](info): commencing operation
krb5kdc: starting...
W20260807 06:52:00.145094 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:52:00 dist-test-slave-3kk6 krb5kdc[15514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:52:00.479240 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40393
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:00.850504 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:00.850889 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:00.851075 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:00.858608 15531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:52:00.858780 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:00.858907 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:00.859017 15531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:52:00.859108 15531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:52:00.869421 15531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:00.871021 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:00.872435 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:00.885033 15538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:00.885207 15537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:00.885447 15540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:00.885906 15531 server_base.cc:1034] running on GCE node
I20260807 06:52:00.886893 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:00.888430 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:00.889642 15531 hybrid_clock.cc:648] HybridClock initialized: now 1786085520889591 us; error 38 us; skew 500 ppm
I20260807 06:52:00.890010 15531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:00 dist-test-slave-3kk6 krb5kdc[15514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085520, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:00.896519 15531 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:52:00.898283 15531 webserver.cc:460] Webserver started at http://127.27.54.126:45139/ using document root <none> and password file <none>
I20260807 06:52:00.899163 15531 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:00.899312 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:00.899682 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:00.903008 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "fb4b695f951446f6b93415a9ab45fdaa"
format_stamp: "Formatted at 2026-08-07 06:52:00 on dist-test-slave-3kk6"
server_key: "0d4e89b35628e00449846883cec4ac7b"
server_key_iv: "d61233bc9fcfdfb626e91785cd2c3c79"
server_key_version: "encryptionkey@0"
I20260807 06:52:00.903877 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "fb4b695f951446f6b93415a9ab45fdaa"
format_stamp: "Formatted at 2026-08-07 06:52:00 on dist-test-slave-3kk6"
server_key: "0d4e89b35628e00449846883cec4ac7b"
server_key_iv: "d61233bc9fcfdfb626e91785cd2c3c79"
server_key_version: "encryptionkey@0"
I20260807 06:52:00.909565 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 06:52:00.913487 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:00.915009 15531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:52:00.915236 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "fb4b695f951446f6b93415a9ab45fdaa"
format_stamp: "Formatted at 2026-08-07 06:52:00 on dist-test-slave-3kk6"
server_key: "0d4e89b35628e00449846883cec4ac7b"
server_key_iv: "d61233bc9fcfdfb626e91785cd2c3c79"
server_key_version: "encryptionkey@0"
I20260807 06:52:00.915491 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:00.928311 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:00.932946 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:00.933351 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:00.953657 15531 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40393
I20260807 06:52:00.953694 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40393 every 8 connection(s)
I20260807 06:52:00.955271 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:52:00.959313 15622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:00.963652 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15531
I20260807 06:52:00.963904 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:52:00.964814 27865 external_mini_cluster.cc:1428] Setting key 2764a3997c02ca2e63ae42a9e4ee8651
I20260807 06:52:00.974174 15622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa: Bootstrap starting.
I20260807 06:52:00.978500 15622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:00.980086 15622 log.cc:824] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:52:00 dist-test-slave-3kk6 krb5kdc[15514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:00.983852 15622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa: No bootstrap required, opened a new log
I20260807 06:52:00.991477 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:00.966982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.27.54.126:40393)
0807 06:52:00.967845 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:00.967857 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:00.967900 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:52:00.969561 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:00.969584 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:00.970031 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:00.970393 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:00.972668 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:00.974056 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:00.976029 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:00.976772 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:00.976916 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:00.982972 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:00.982981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:00.983004 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:00.983055 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:00.986522 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:00.987591 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:00.987599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:00.987609 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:00.987839 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:00.988293 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:00.988296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:00.988298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:00.988901 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:00.989159 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:00.989775 (+ 616us) server_negotiation.cc:299] Negotiation successful
0807 06:52:00.990266 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":412,"threads_started":1}
I20260807 06:52:00.993727 15622 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4b695f951446f6b93415a9ab45fdaa" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40393 } }
I20260807 06:52:00.994211 15622 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:00.994346 15622 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4b695f951446f6b93415a9ab45fdaa, State: Initialized, Role: FOLLOWER
I20260807 06:52:00.995114 15622 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4b695f951446f6b93415a9ab45fdaa" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40393 } }
I20260807 06:52:00.995393 15622 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:52:00.995564 15622 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:52:00.995756 15622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:00.996155 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:52:00.999274 15622 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4b695f951446f6b93415a9ab45fdaa" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40393 } }
I20260807 06:52:00.999871 15622 leader_election.cc:302] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4b695f951446f6b93415a9ab45fdaa; no voters:
I20260807 06:52:01.000590 15622 leader_election.cc:288] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:52:01.000747 15631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:01.001515 15631 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [term 1 LEADER]: Becoming Leader. State: Replica: fb4b695f951446f6b93415a9ab45fdaa, State: Running, Role: LEADER
I20260807 06:52:01.002085 15631 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4b695f951446f6b93415a9ab45fdaa" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40393 } }
I20260807 06:52:01.002878 15622 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:52:01.005658 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb4b695f951446f6b93415a9ab45fdaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4b695f951446f6b93415a9ab45fdaa" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40393 } } }
I20260807 06:52:01.005635 15634 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb4b695f951446f6b93415a9ab45fdaa. Latest consensus state: current_term: 1 leader_uuid: "fb4b695f951446f6b93415a9ab45fdaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4b695f951446f6b93415a9ab45fdaa" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40393 } } }
I20260807 06:52:01.006162 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:01.006181 15634 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:01.008239 15639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:52:01.015012 15639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:52:01.025050 15639 catalog_manager.cc:1347] Generated new cluster ID: 02fbe41ce57443a3a6ae6f442206808b
I20260807 06:52:01.025197 15639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:52:01.056397 15639 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:52:01.057839 15639 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:52:01.070395 15639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb4b695f951446f6b93415a9ab45fdaa: Generated new TSK 0
I20260807 06:52:01.071367 15639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:52:01.111312 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15531
I20260807 06:52:01.112979 15614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:01.214252 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15531
2026-08-07T06:52:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:52:01 dist-test-slave-3kk6 krb5kdc[15664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:52:01 dist-test-slave-3kk6 krb5kdc[15664](info): set up 2 sockets
Aug 07 06:52:01 dist-test-slave-3kk6 krb5kdc[15664](info): commencing operation
krb5kdc: starting...
W20260807 06:52:02.648975 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.384s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:52:02 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:52:02.974746 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:35595
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:03.329561 15680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:03.329936 15680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:03.330102 15680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:03.337505 15680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:52:03.337620 15680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:03.337776 15680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:03.337879 15680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:52:03.337970 15680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:52:03.348223 15680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:35595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:03.349658 15680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:03.351052 15680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:03.363301 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:03.363343 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:03.363255 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:03.364557 15680 server_base.cc:1034] running on GCE node
I20260807 06:52:03.365355 15680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:03.366851 15680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:03.368088 15680 hybrid_clock.cc:648] HybridClock initialized: now 1786085523368051 us; error 47 us; skew 500 ppm
I20260807 06:52:03.368462 15680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:03 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085523, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:03.371779 15680 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:52:03.373404 15680 webserver.cc:460] Webserver started at http://127.27.54.126:44173/ using document root <none> and password file <none>
I20260807 06:52:03.374120 15680 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:03.374279 15680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:03.374608 15680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:03.377724 15680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "d60b421756234074958a269dfd67caa3"
format_stamp: "Formatted at 2026-08-07 06:52:03 on dist-test-slave-3kk6"
server_key: "35ffc6af7f7ef15f0401ec0bfad5dce9"
server_key_iv: "c93717001ec8f54d0b5b9f1a9f9f548e"
server_key_version: "encryptionkey@0"
I20260807 06:52:03.378564 15680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "d60b421756234074958a269dfd67caa3"
format_stamp: "Formatted at 2026-08-07 06:52:03 on dist-test-slave-3kk6"
server_key: "35ffc6af7f7ef15f0401ec0bfad5dce9"
server_key_iv: "c93717001ec8f54d0b5b9f1a9f9f548e"
server_key_version: "encryptionkey@0"
I20260807 06:52:03.384109 15680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:52:03.387704 15700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:03.389026 15680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:52:03.389231 15680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "d60b421756234074958a269dfd67caa3"
format_stamp: "Formatted at 2026-08-07 06:52:03 on dist-test-slave-3kk6"
server_key: "35ffc6af7f7ef15f0401ec0bfad5dce9"
server_key_iv: "c93717001ec8f54d0b5b9f1a9f9f548e"
server_key_version: "encryptionkey@0"
I20260807 06:52:03.389451 15680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:03.403981 15680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:03.408054 15680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:03.408360 15680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:03.427907 15680 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:35595
I20260807 06:52:03.427955 15766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:35595 every 8 connection(s)
I20260807 06:52:03.429390 15680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:52:03.433288 15767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:03.437201 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15680
I20260807 06:52:03.437462 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:52:03.438333 27865 external_mini_cluster.cc:1428] Setting key 1fd5ec855554db752e2bc621d0fff6c3
I20260807 06:52:03.447361 15767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap starting.
I20260807 06:52:03.451243 15767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:03.452606 15767 log.cc:824] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Log is configured to *not* fsync() on all Append() calls
Aug 07 06:52:03 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:03.455883 15767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: No bootstrap required, opened a new log
I20260807 06:52:03.464344 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:03.440591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.27.54.126:35595)
0807 06:52:03.441418 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:03.441431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:03.441471 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:52:03.443149 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:03.443171 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:03.443560 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:03.443899 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:03.445891 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:03.447301 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:03.449194 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:03.449867 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:03.449998 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:03.455199 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:03.455214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:03.455236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:03.455291 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:03.459502 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:03.460233 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:03.460250 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:03.460259 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:03.460484 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:03.460883 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:03.460887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:03.460890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:03.461668 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:03.462008 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:03.462383 (+ 375us) server_negotiation.cc:299] Negotiation successful
0807 06:52:03.462879 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":399,"threads_started":1}
I20260807 06:52:03.467407 15767 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:03.467829 15767 raft_consensus.cc:374] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:03.467921 15767 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60b421756234074958a269dfd67caa3, State: Initialized, Role: FOLLOWER
I20260807 06:52:03.468551 15767 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:03.468827 15767 raft_consensus.cc:388] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:52:03.468952 15767 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:52:03.469099 15767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:03.472234 15767 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:03.472818 15767 leader_election.cc:302] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d60b421756234074958a269dfd67caa3; no voters:
I20260807 06:52:03.473507 15767 leader_election.cc:288] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:52:03.473701 15773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:03.477262 15767 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:52:03.477450 15773 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 LEADER]: Becoming Leader. State: Replica: d60b421756234074958a269dfd67caa3, State: Running, Role: LEADER
I20260807 06:52:03.478120 15773 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:03.481822 15777 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60b421756234074958a269dfd67caa3. Latest consensus state: current_term: 1 leader_uuid: "d60b421756234074958a269dfd67caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } } }
I20260807 06:52:03.481792 15774 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d60b421756234074958a269dfd67caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } } }
I20260807 06:52:03.482295 15777 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:03.482295 15774 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:03.483309 15793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:52:03.488868 15793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:52:03.498274 15793 catalog_manager.cc:1347] Generated new cluster ID: 2b2bb0249da24564b9ecb8abb0d80b84
I20260807 06:52:03.498413 15793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:52:03.516441 15793 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:52:03.517797 15793 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:52:03.527570 15793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Generated new TSK 0
I20260807 06:52:03.528421 15793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:52:03.606066 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:35595
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:03.959884 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:03.960172 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:03.960369 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:03.970475 15807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:03.970629 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:03.970847 15807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:52:03.981499 15807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:03.982990 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:03.984354 15807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:03.998121 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:03.998081 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:03.998159 15815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:03.998996 15807 server_base.cc:1034] running on GCE node
I20260807 06:52:03.999675 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:04.000648 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:04.001886 15807 hybrid_clock.cc:648] HybridClock initialized: now 1786085524001838 us; error 45 us; skew 500 ppm
I20260807 06:52:04.002223 15807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:04 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:04.005625 15807 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:52:04.007159 15807 webserver.cc:460] Webserver started at http://127.27.54.65:37335/ using document root <none> and password file <none>
I20260807 06:52:04.007814 15807 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:04.007918 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:04.008172 15807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:04.011178 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "5bf89563309345ba98ecdcb5d15aba99"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "538fcf99a25694ae710682107d9a6af3"
server_key_iv: "b09e1aa3a6c78a9d7f344fadf65c4211"
server_key_version: "encryptionkey@0"
I20260807 06:52:04.011982 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf89563309345ba98ecdcb5d15aba99"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "538fcf99a25694ae710682107d9a6af3"
server_key_iv: "b09e1aa3a6c78a9d7f344fadf65c4211"
server_key_version: "encryptionkey@0"
I20260807 06:52:04.017244 15807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 06:52:04.020658 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:04.021974 15807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:52:04.022131 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "5bf89563309345ba98ecdcb5d15aba99"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "538fcf99a25694ae710682107d9a6af3"
server_key_iv: "b09e1aa3a6c78a9d7f344fadf65c4211"
server_key_version: "encryptionkey@0"
I20260807 06:52:04.022306 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:04.041399 15807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:04.046164 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:04.046499 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:04.047691 15807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:04.049103 15807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:04.049212 15807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:04.049373 15807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:04.049450 15807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:04.084389 15807 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43887
I20260807 06:52:04.084427 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43887 every 8 connection(s)
I20260807 06:52:04.085729 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:52:04.088240 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15807
I20260807 06:52:04.088517 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:52:04.089232 27865 external_mini_cluster.cc:1428] Setting key 79a5e5b3887cbe845b2ca83a57b040d9
Aug 07 06:52:04 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:04.111672 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:04.088423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51845 (local address 127.27.54.126:35595)
0807 06:52:04.090964 (+ 2541us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:04.090969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:04.090988 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:52:04.092715 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:04.092723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:04.092823 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:04.092968 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:04.094868 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.096032 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.098992 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.099977 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.100127 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:04.105462 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:04.105469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:04.105476 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:04.105523 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:04.108374 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:04.109158 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:04.109165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:04.109173 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:04.109300 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:04.109780 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:04.109783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:04.109786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:04.110059 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:04.110222 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:04.111042 (+ 820us) server_negotiation.cc:299] Negotiation successful
0807 06:52:04.111278 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2434,"thread_start_us":2319,"threads_started":1}
I20260807 06:52:04.112700 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:04.089080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:51845)
0807 06:52:04.090126 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:04.090193 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:04.092425 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:04.093207 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:04.093215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:04.093893 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:04.094608 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:04.094630 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.096281 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.096288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:04.098792 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:04.098813 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.100481 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.100491 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:04.100849 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:04.101733 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:04.101793 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:04.105147 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:04.108543 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:04.108552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:04.108569 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:04.109016 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:04.109423 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:04.109428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:04.109432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:04.109640 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:04.110201 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:04.110209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:04.110575 (+ 366us) client_negotiation.cc:769] Sending connection context
0807 06:52:04.110931 (+ 356us) client_negotiation.cc:240] Negotiation successful
0807 06:52:04.111506 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":393,"threads_started":1}
I20260807 06:52:04.114400 15974 heartbeater.cc:347] Connected to a master server at 127.27.54.126:35595
I20260807 06:52:04.114822 15974 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:04.115767 15974 heartbeater.cc:511] Master 127.27.54.126:35595 requested a full tablet report, sending...
I20260807 06:52:04.118408 15722 ts_manager.cc:194] Registered new tserver with Master: 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65:43887)
I20260807 06:52:04.120702 15722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:51845
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:52:04.162197 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:35595
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:04.513453 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:04.513795 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:04.513967 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:04.521296 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:04.521412 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:04.521584 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:52:04.531761 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:04.533138 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:04.534495 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:04.548375 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:04.548496 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:04.548542 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:04.549204 15984 server_base.cc:1034] running on GCE node
I20260807 06:52:04.549842 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:04.550851 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:04.552062 15984 hybrid_clock.cc:648] HybridClock initialized: now 1786085524552003 us; error 53 us; skew 500 ppm
I20260807 06:52:04.552356 15984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:04 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:04.556093 15984 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:52:04.557708 15984 webserver.cc:460] Webserver started at http://127.27.54.66:40649/ using document root <none> and password file <none>
I20260807 06:52:04.558473 15984 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:04.558616 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:04.559010 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:04.562188 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "98bcf00a9ad8488784ab75c777a1a57c"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "88d9c6f372fe0cb01fc7d5a28d09508d"
server_key_iv: "8939ee45dd63d85a0ba124acdf1a7ae4"
server_key_version: "encryptionkey@0"
I20260807 06:52:04.563126 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "98bcf00a9ad8488784ab75c777a1a57c"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "88d9c6f372fe0cb01fc7d5a28d09508d"
server_key_iv: "8939ee45dd63d85a0ba124acdf1a7ae4"
server_key_version: "encryptionkey@0"
I20260807 06:52:04.568612 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:52:04.572266 16009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:04.573730 15984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:52:04.573930 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "98bcf00a9ad8488784ab75c777a1a57c"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "88d9c6f372fe0cb01fc7d5a28d09508d"
server_key_iv: "8939ee45dd63d85a0ba124acdf1a7ae4"
server_key_version: "encryptionkey@0"
I20260807 06:52:04.574167 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:04.590160 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:04.595145 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:04.595474 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:04.596557 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:04.598014 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:04.598124 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:04.598282 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:04.598371 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:04.634044 15984 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:44095
I20260807 06:52:04.634084 16159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:44095 every 8 connection(s)
I20260807 06:52:04.635488 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:52:04.644187 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 15984
I20260807 06:52:04.644405 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:52:04.644958 27865 external_mini_cluster.cc:1428] Setting key a2f3ecd958d4269a35edff88a7237aa7
Aug 07 06:52:04 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:04.657768 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:04.638073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:60335 (local address 127.27.54.126:35595)
0807 06:52:04.638561 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:04.638569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:04.639629 (+ 1060us) server_negotiation.cc:407] Connection header received
0807 06:52:04.641439 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:04.641444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:04.641518 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:04.641623 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:04.643326 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.644260 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.646101 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.646553 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.646724 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:04.650402 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:04.650410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:04.650418 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:04.650466 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:04.654197 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:04.655059 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:04.655068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:04.655074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:04.655267 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:04.655704 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:04.655712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:04.655718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:04.656120 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:04.656295 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:04.656996 (+ 701us) server_negotiation.cc:299] Negotiation successful
0807 06:52:04.657266 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260807 06:52:04.658416 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:04.638724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:60335)
0807 06:52:04.639460 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:04.639505 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:04.641153 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:04.641854 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:04.641862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:04.642371 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:04.643126 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:04.643158 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.644499 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.644507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:04.645914 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:04.645928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:04.646822 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:04.646830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:04.647067 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:04.647804 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:04.647828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:04.650129 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:04.654374 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:04.654381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:04.654396 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:04.654914 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:04.655385 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:04.655388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:04.655391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:04.655580 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:04.656261 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:04.656267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:04.656598 (+ 331us) client_negotiation.cc:769] Sending connection context
0807 06:52:04.656835 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 06:52:04.657259 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":291,"threads_started":1}
I20260807 06:52:04.660022 16161 heartbeater.cc:347] Connected to a master server at 127.27.54.126:35595
I20260807 06:52:04.660370 16161 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:04.661229 16161 heartbeater.cc:511] Master 127.27.54.126:35595 requested a full tablet report, sending...
I20260807 06:52:04.662935 15722 ts_manager.cc:194] Registered new tserver with Master: 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095)
I20260807 06:52:04.663861 15722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:60335
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:52:04.714298 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:35595
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:05.066320 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:05.066625 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:05.066820 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:05.074159 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:05.074309 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:05.074512 16171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:52:05.085175 16171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:05.086441 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:05.087848 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:05.101338 16188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:05.101486 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:05.101457 16184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:05.102461 16171 server_base.cc:1034] running on GCE node
I20260807 06:52:05.103169 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:05.104187 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:05.105396 16171 hybrid_clock.cc:648] HybridClock initialized: now 1786085525105354 us; error 43 us; skew 500 ppm
I20260807 06:52:05.105763 16171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:05 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085525, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:05.109294 16171 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:52:05.110980 16171 webserver.cc:460] Webserver started at http://127.27.54.67:45461/ using document root <none> and password file <none>
I20260807 06:52:05.111721 16171 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:05.111872 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:05.112205 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:05.115332 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "98bfe8095c6f4ba7be8f72954105f985"
format_stamp: "Formatted at 2026-08-07 06:52:05 on dist-test-slave-3kk6"
server_key: "8f958298132ac4baa82ac028ba2e825d"
server_key_iv: "302a2e68c50f4dc0ff55c5d5fff7c9a9"
server_key_version: "encryptionkey@0"
I20260807 06:52:05.116214 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "98bfe8095c6f4ba7be8f72954105f985"
format_stamp: "Formatted at 2026-08-07 06:52:05 on dist-test-slave-3kk6"
server_key: "8f958298132ac4baa82ac028ba2e825d"
server_key_iv: "302a2e68c50f4dc0ff55c5d5fff7c9a9"
server_key_version: "encryptionkey@0"
I20260807 06:52:05.121507 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 06:52:05.124727 15974 heartbeater.cc:503] Master 127.27.54.126:35595 was elected leader, sending a full tablet report...
I20260807 06:52:05.125422 16198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:05.127113 16171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:52:05.127301 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "98bfe8095c6f4ba7be8f72954105f985"
format_stamp: "Formatted at 2026-08-07 06:52:05 on dist-test-slave-3kk6"
server_key: "8f958298132ac4baa82ac028ba2e825d"
server_key_iv: "302a2e68c50f4dc0ff55c5d5fff7c9a9"
server_key_version: "encryptionkey@0"
I20260807 06:52:05.127511 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:05.151518 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:05.155833 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:05.156169 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:05.157199 16171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:05.158612 16171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:05.158737 16171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 06:52:05.158907 16171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:05.158983 16171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:05.194307 16171 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:46351
I20260807 06:52:05.194404 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:46351 every 8 connection(s)
I20260807 06:52:05.195781 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:52:05.206213 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 16171
I20260807 06:52:05.206459 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:52:05.207234 27865 external_mini_cluster.cc:1428] Setting key a5bfa8b23900ee908200ea029004a877
Aug 07 06:52:05 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085525, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:05.221369 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.198450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:54691 (local address 127.27.54.126:35595)
0807 06:52:05.199004 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.199009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.199863 (+ 854us) server_negotiation.cc:407] Connection header received
0807 06:52:05.202042 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.202049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.202178 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.202324 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:05.204419 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.205731 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.207815 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.208426 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.209598 (+ 1172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.213694 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:05.213698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:05.213702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:05.213765 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:05.217345 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.218468 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.218472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.218474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.218645 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.219261 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.219265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.219267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.219550 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:05.219673 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.220768 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.220981 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":316,"threads_started":1}
I20260807 06:52:05.222913 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.198951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:54691)
0807 06:52:05.199687 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:05.199729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:05.201684 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:05.202583 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:05.202596 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:05.203450 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:05.204181 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.204200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.205902 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.205907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.207654 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.207674 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.208775 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.208786 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.209160 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.210007 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:05.210043 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:05.213433 (+ 3390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:05.217521 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:05.217532 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:05.217552 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:05.218313 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:05.218807 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:05.218813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:05.218816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:05.219113 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:05.219712 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:05.219723 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:05.220268 (+ 545us) client_negotiation.cc:769] Sending connection context
0807 06:52:05.220607 (+ 339us) client_negotiation.cc:240] Negotiation successful
0807 06:52:05.221316 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":300,"threads_started":1}
I20260807 06:52:05.224429 16351 heartbeater.cc:347] Connected to a master server at 127.27.54.126:35595
I20260807 06:52:05.224778 16351 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:05.225598 16351 heartbeater.cc:511] Master 127.27.54.126:35595 requested a full tablet report, sending...
I20260807 06:52:05.227186 15722 ts_manager.cc:194] Registered new tserver with Master: 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67:46351)
I20260807 06:52:05.228039 15722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:54691
I20260807 06:52:05.236348 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:52:05.258559 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.245701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.27.54.126:35595)
0807 06:52:05.245955 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.245961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.246241 (+ 280us) server_negotiation.cc:407] Connection header received
0807 06:52:05.246402 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.246406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.246471 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.246568 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:05.247989 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.248770 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.250243 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.250636 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.250800 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.253214 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:05.253219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:05.253222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:05.253254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:05.256010 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.256661 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.256666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.256669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.256790 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.257176 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.257181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.257183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.257529 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:05.257721 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.258007 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.258177 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:52:05.262632 15722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:52:05.269924 15722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:52:05.310142 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.283890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43887 (local address 127.0.0.1:48844)
0807 06:52:05.301709 (+ 17819us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:05.301728 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:05.301886 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:05.302702 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:05.302708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:05.302741 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:05.303138 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.303151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.304983 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.304991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.307052 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.307065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.309036 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.309043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.309251 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.309311 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 06:52:05.309434 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 06:52:05.309626 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17680,"mutex_wait_us":1,"thread_start_us":17115,"threads_started":1}
I20260807 06:52:05.310766 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.283070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:44095 (local address 127.0.0.1:41928)
0807 06:52:05.298295 (+ 15225us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:05.298354 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:05.298632 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:05.301750 (+ 3118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:05.301761 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:05.301799 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:05.302263 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.302294 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.304331 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.304338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.307545 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.307561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.309836 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.309843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.310007 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.310039 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:52:05.310169 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 06:52:05.310309 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14882,"mutex_wait_us":48,"thread_start_us":17363,"threads_started":1}
I20260807 06:52:05.311025 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.284698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.27.54.65:43887)
0807 06:52:05.292132 (+ 7434us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.292146 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.302050 (+ 9904us) server_negotiation.cc:407] Connection header received
0807 06:52:05.302172 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.302182 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.302471 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.302698 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:05.303332 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.304823 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.307215 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.308191 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.308646 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.308900 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.310451 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.310673 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7200,"thread_start_us":6968,"threads_started":1}
I20260807 06:52:05.311226 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.283285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.27.54.66:44095)
0807 06:52:05.299343 (+ 16058us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.299355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.299397 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:52:05.299542 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.299554 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.299906 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.300167 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:05.302519 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.304180 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.307738 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.309599 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.309777 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.309972 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.310225 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.310586 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15902,"thread_start_us":15768,"threads_started":1}
I20260807 06:52:05.311789 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.283991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.27.54.67:46351)
0807 06:52:05.303000 (+ 19009us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.303011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.303048 (+ 37us) server_negotiation.cc:407] Connection header received
0807 06:52:05.303151 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.303158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.303377 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.303582 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:05.304212 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.305838 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.308515 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.309915 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.310054 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.311226 (+ 1172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.311300 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.311487 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18817,"thread_start_us":19819,"threads_started":1}
I20260807 06:52:05.312507 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.283337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:46351 (local address 127.0.0.1:50440)
0807 06:52:05.301310 (+ 17973us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:05.301334 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:05.301459 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:05.303567 (+ 2108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:05.303572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:05.303598 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:05.304020 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.304032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.306020 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.306025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.308321 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.308336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.310790 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.310799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.310970 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.311007 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:52:05.311999 (+ 992us) client_negotiation.cc:240] Negotiation successful
0807 06:52:05.312147 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17822,"mutex_wait_us":24,"thread_start_us":18071,"threads_started":1}
I20260807 06:52:05.315193 15891 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9b9942f1c84aeba1e0b71befcbf250 (DEFAULT_TABLE table=test-table [id=2a99219150104a5b9164a4dff558131d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:05.315722 16259 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9b9942f1c84aeba1e0b71befcbf250 (DEFAULT_TABLE table=test-table [id=2a99219150104a5b9164a4dff558131d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:05.316865 15891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9b9942f1c84aeba1e0b71befcbf250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:05.316969 16259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9b9942f1c84aeba1e0b71befcbf250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:05.317236 16071 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9b9942f1c84aeba1e0b71befcbf250 (DEFAULT_TABLE table=test-table [id=2a99219150104a5b9164a4dff558131d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:05.318529 16071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9b9942f1c84aeba1e0b71befcbf250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:05.328794 16380 tablet_bootstrap.cc:492] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Bootstrap starting.
I20260807 06:52:05.332140 16381 tablet_bootstrap.cc:492] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Bootstrap starting.
I20260807 06:52:05.333508 16384 tablet_bootstrap.cc:492] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Bootstrap starting.
I20260807 06:52:05.334277 16380 tablet_bootstrap.cc:654] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:05.335685 16380 log.cc:824] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:05.336035 16381 tablet_bootstrap.cc:654] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:05.336689 16384 tablet_bootstrap.cc:654] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:05.337240 16381 log.cc:824] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:05.337915 16384 log.cc:824] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:05.339303 16380 tablet_bootstrap.cc:492] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: No bootstrap required, opened a new log
I20260807 06:52:05.339576 16380 ts_tablet_manager.cc:1397] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260807 06:52:05.340471 16381 tablet_bootstrap.cc:492] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: No bootstrap required, opened a new log
I20260807 06:52:05.340754 16381 ts_tablet_manager.cc:1397] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260807 06:52:05.341015 16384 tablet_bootstrap.cc:492] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: No bootstrap required, opened a new log
I20260807 06:52:05.341271 16384 ts_tablet_manager.cc:1397] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 06:52:05.347952 16380 raft_consensus.cc:348] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.348351 16380 raft_consensus.cc:374] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:05.348476 16380 raft_consensus.cc:729] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bfe8095c6f4ba7be8f72954105f985, State: Initialized, Role: FOLLOWER
I20260807 06:52:05.349107 16380 consensus_queue.cc:260] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.350229 16351 heartbeater.cc:503] Master 127.27.54.126:35595 was elected leader, sending a full tablet report...
I20260807 06:52:05.351025 16380 ts_tablet_manager.cc:1428] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260807 06:52:05.351575 16381 raft_consensus.cc:348] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.352144 16381 raft_consensus.cc:374] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:05.351900 16384 raft_consensus.cc:348] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.352254 16381 raft_consensus.cc:729] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf89563309345ba98ecdcb5d15aba99, State: Initialized, Role: FOLLOWER
I20260807 06:52:05.352336 16384 raft_consensus.cc:374] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:05.352442 16384 raft_consensus.cc:729] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bcf00a9ad8488784ab75c777a1a57c, State: Initialized, Role: FOLLOWER
I20260807 06:52:05.352895 16381 consensus_queue.cc:260] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.352932 16384 consensus_queue.cc:260] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.354853 16381 ts_tablet_manager.cc:1428] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260807 06:52:05.355158 16384 ts_tablet_manager.cc:1428] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260807 06:52:05.355207 16161 heartbeater.cc:503] Master 127.27.54.126:35595 was elected leader, sending a full tablet report...
I20260807 06:52:05.371448 16391 raft_consensus.cc:482] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:52:05.371774 16391 raft_consensus.cc:504] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.373482 16391 leader_election.cc:288] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095), 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65:43887)
I20260807 06:52:05.380254 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.373966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:45797 (local address 127.27.54.65:43887)
0807 06:52:05.374182 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.374188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.374374 (+ 186us) server_negotiation.cc:407] Connection header received
0807 06:52:05.374541 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.374549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.374668 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.374792 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:05.375307 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.376256 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.378404 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.379334 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.379503 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.379575 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.379863 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.379990 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:52:05.380288 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.374027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43887 (local address 127.27.54.67:45797)
0807 06:52:05.374250 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:05.374271 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:05.374398 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:05.374826 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:05.374833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:05.374880 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:05.375128 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.375140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.376387 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.376390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.378243 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.378258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.379500 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.379507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.379700 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.379727 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:52:05.379836 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 06:52:05.379965 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 06:52:05.381086 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.374035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:55529 (local address 127.27.54.66:44095)
0807 06:52:05.374231 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.374237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.375303 (+ 1066us) server_negotiation.cc:407] Connection header received
0807 06:52:05.375457 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.375462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.375537 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.375639 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:05.376200 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.376956 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.379152 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.379906 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.380092 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.380179 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.380598 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.380793 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:52:05.381448 15915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b9942f1c84aeba1e0b71befcbf250" candidate_uuid: "98bfe8095c6f4ba7be8f72954105f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf89563309345ba98ecdcb5d15aba99" is_pre_election: true
I20260807 06:52:05.381881 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.374023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:44095 (local address 127.27.54.67:55529)
0807 06:52:05.375187 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:05.375205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:05.375322 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:05.375654 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:05.375656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:05.375674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:05.376034 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.376042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.377106 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.377112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.378959 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:05.378971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.380112 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.380136 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:05.380315 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.380423 (+ 108us) client_negotiation.cc:769] Sending connection context
0807 06:52:05.381289 (+ 866us) client_negotiation.cc:240] Negotiation successful
0807 06:52:05.381419 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":189,"thread_start_us":570,"threads_started":1}
I20260807 06:52:05.382158 15915 raft_consensus.cc:2454] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98bfe8095c6f4ba7be8f72954105f985 in term 0.
I20260807 06:52:05.383013 16203 leader_election.cc:302] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf89563309345ba98ecdcb5d15aba99, 98bfe8095c6f4ba7be8f72954105f985; no voters:
I20260807 06:52:05.383054 16100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b9942f1c84aeba1e0b71befcbf250" candidate_uuid: "98bfe8095c6f4ba7be8f72954105f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bcf00a9ad8488784ab75c777a1a57c" is_pre_election: true
I20260807 06:52:05.383486 16391 raft_consensus.cc:2781] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:52:05.383638 16391 raft_consensus.cc:482] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:52:05.383625 16100 raft_consensus.cc:2454] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98bfe8095c6f4ba7be8f72954105f985 in term 0.
I20260807 06:52:05.383769 16391 raft_consensus.cc:3037] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:05.386639 16391 raft_consensus.cc:504] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.387377 16391 leader_election.cc:288] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [CANDIDATE]: Term 1 election: Requested vote from peers 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095), 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65:43887)
W20260807 06:52:05.387590 16162 tablet.cc:2369] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:05.387914 15915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b9942f1c84aeba1e0b71befcbf250" candidate_uuid: "98bfe8095c6f4ba7be8f72954105f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf89563309345ba98ecdcb5d15aba99"
I20260807 06:52:05.387898 16100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b9942f1c84aeba1e0b71befcbf250" candidate_uuid: "98bfe8095c6f4ba7be8f72954105f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bcf00a9ad8488784ab75c777a1a57c"
I20260807 06:52:05.388131 16100 raft_consensus.cc:3037] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:05.388132 15915 raft_consensus.cc:3037] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:05.390369 15915 raft_consensus.cc:2454] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98bfe8095c6f4ba7be8f72954105f985 in term 1.
I20260807 06:52:05.390388 16100 raft_consensus.cc:2454] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98bfe8095c6f4ba7be8f72954105f985 in term 1.
I20260807 06:52:05.390805 16203 leader_election.cc:302] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf89563309345ba98ecdcb5d15aba99, 98bfe8095c6f4ba7be8f72954105f985; no voters:
I20260807 06:52:05.391095 16391 raft_consensus.cc:2781] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:05.391539 16391 raft_consensus.cc:686] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 1 LEADER]: Becoming Leader. State: Replica: 98bfe8095c6f4ba7be8f72954105f985, State: Running, Role: LEADER
I20260807 06:52:05.392112 16391 consensus_queue.cc:237] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } }
I20260807 06:52:05.397300 15722 catalog_manager.cc:5495] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67). New cstate: current_term: 1 leader_uuid: "98bfe8095c6f4ba7be8f72954105f985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } health_report { overall_health: UNKNOWN } } }
I20260807 06:52:05.417006 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.411194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.27.54.67:46351)
0807 06:52:05.411408 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.411416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.411436 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:52:05.411722 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.411726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.411817 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.411988 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:52:05.412385 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.413124 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.414800 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.415207 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.415348 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.415883 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:52:05.416388 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:52:05.416547 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.416647 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.416740 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:52:05.425587 16100 raft_consensus.cc:1261] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Refusing update from remote peer 98bfe8095c6f4ba7be8f72954105f985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:05.425626 15915 raft_consensus.cc:1261] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Refusing update from remote peer 98bfe8095c6f4ba7be8f72954105f985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:05.426689 16395 consensus_queue.cc:1035] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:52:05.427135 16391 consensus_queue.cc:1035] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:52:05.443420 16399 mvcc.cc:205] Tried to move back new op lower bound from 7315806312131993600 to 7315806312010539008. Current Snapshot: MvccSnapshot[applied={T|T < 7315806312131993600}]
I20260807 06:52:05.446678 16402 mvcc.cc:205] Tried to move back new op lower bound from 7315806312131993600 to 7315806312010539008. Current Snapshot: MvccSnapshot[applied={T|T < 7315806312131993600}]
I20260807 06:52:05.451571 16400 mvcc.cc:205] Tried to move back new op lower bound from 7315806312131993600 to 7315806312010539008. Current Snapshot: MvccSnapshot[applied={T|T < 7315806312131993600}]
I20260807 06:52:05.466171 15722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:05.466495 15722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:05.470561 15722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Sending DeleteTablet for 3 replicas of tablet 0a9b9942f1c84aeba1e0b71befcbf250
I20260807 06:52:05.471849 16259 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9b9942f1c84aeba1e0b71befcbf250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:05 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:50440
I20260807 06:52:05.471961 15891 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9b9942f1c84aeba1e0b71befcbf250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:05 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:48844
I20260807 06:52:05.471972 16071 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9b9942f1c84aeba1e0b71befcbf250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:05 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:41928
I20260807 06:52:05.473311 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15807
I20260807 06:52:05.485545 16414 tablet_replica.cc:330] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: stopping tablet replica
I20260807 06:52:05.486189 16414 raft_consensus.cc:2229] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:05.487169 16413 tablet_replica.cc:330] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: stopping tablet replica
I20260807 06:52:05.487650 16414 raft_consensus.cc:2258] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:05.487645 16413 raft_consensus.cc:2229] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:52:05.488349 16413 raft_consensus.cc:2258] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:05.490864 16413 ts_tablet_manager.cc:1905] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:05.492166 16412 tablet_replica.cc:330] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: stopping tablet replica
I20260807 06:52:05.492862 16412 raft_consensus.cc:2229] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:05.493590 16412 raft_consensus.cc:2258] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:05.496059 16412 ts_tablet_manager.cc:1905] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 06:52:05 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:52:05.498420 16414 ts_tablet_manager.cc:1905] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:05.499253 16413 ts_tablet_manager.cc:1918] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:05.499461 16413 log.cc:1196] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/0a9b9942f1c84aeba1e0b71befcbf250
I20260807 06:52:05.500048 16413 ts_tablet_manager.cc:1939] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bfe8095c6f4ba7be8f72954105f985: Deleting consensus metadata
I20260807 06:52:05.502020 15704 catalog_manager.cc:4841] TS 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67:46351): tablet 0a9b9942f1c84aeba1e0b71befcbf250 (table test-table [id=2a99219150104a5b9164a4dff558131d]) successfully deleted
I20260807 06:52:05.503983 16412 ts_tablet_manager.cc:1918] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:05.504274 16412 log.cc:1196] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/0a9b9942f1c84aeba1e0b71befcbf250
I20260807 06:52:05.504958 16412 ts_tablet_manager.cc:1939] T 0a9b9942f1c84aeba1e0b71befcbf250 P 5bf89563309345ba98ecdcb5d15aba99: Deleting consensus metadata
I20260807 06:52:05.505813 16414 ts_tablet_manager.cc:1918] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:05.505990 16414 log.cc:1196] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/0a9b9942f1c84aeba1e0b71befcbf250
I20260807 06:52:05.506604 16414 ts_tablet_manager.cc:1939] T 0a9b9942f1c84aeba1e0b71befcbf250 P 98bcf00a9ad8488784ab75c777a1a57c: Deleting consensus metadata
I20260807 06:52:05.507246 15705 catalog_manager.cc:4841] TS 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65:43887): tablet 0a9b9942f1c84aeba1e0b71befcbf250 (table test-table [id=2a99219150104a5b9164a4dff558131d]) successfully deleted
I20260807 06:52:05.508445 15702 catalog_manager.cc:4841] TS 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095): tablet 0a9b9942f1c84aeba1e0b71befcbf250 (table test-table [id=2a99219150104a5b9164a4dff558131d]) successfully deleted
I20260807 06:52:05.511719 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.474631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.27.54.65:43887)
0807 06:52:05.474851 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.474858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.482192 (+ 7334us) server_negotiation.cc:407] Connection header received
0807 06:52:05.482458 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.482464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.482556 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.482696 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:05.487344 (+ 4648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.488540 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.491047 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.491763 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.491887 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.503915 (+ 12028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:05.503934 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:05.503955 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:05.504009 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:05.508380 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.509445 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.509453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.509456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.509601 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.510064 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.510067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.510069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.510443 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:05.510761 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.511160 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.511367 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 06:52:05.512480 15967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:05.617147 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15807
I20260807 06:52:05.648612 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15984
Aug 07 06:52:05 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:52:05.667411 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.649447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.27.54.66:44095)
0807 06:52:05.649647 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.649653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.649740 (+ 87us) server_negotiation.cc:407] Connection header received
0807 06:52:05.649958 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.649964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.650048 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.650189 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:05.651937 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.653141 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.654955 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.655592 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.655729 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.660130 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:05.660147 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:05.660176 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:05.660220 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:05.664253 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.665080 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.665090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.665094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.665262 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.665660 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.665666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.665669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.666186 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:05.666483 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.666789 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.667027 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:52:05.668202 16153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:05.762063 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15984
I20260807 06:52:05.794572 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 16171
Aug 07 06:52:05 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:52:05.815307 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:05.795488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.27.54.67:46351)
0807 06:52:05.795663 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:05.795668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:05.795712 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:52:05.795962 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:05.795965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:05.796029 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:05.796112 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:05.798042 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.799009 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.801014 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:05.801464 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:05.801572 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:05.807587 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:05.807607 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:05.807630 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:05.807681 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:05.812114 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.812933 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.812941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.812944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.813117 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:05.813543 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:05.813549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:05.813554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:05.813917 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:05.814325 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:05.814617 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 06:52:05.814906 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:52:05.816042 16343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:05.941829 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 16171
I20260807 06:52:05.967722 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 15680
I20260807 06:52:05.968909 15762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:06.083428 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 15680
I20260807 06:52:06.115152 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:35595
--webserver_interface=127.27.54.126
--webserver_port=44173
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:06.476974 16422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:06.477315 16422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:06.477515 16422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:06.484921 16422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:52:06.485059 16422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:06.485153 16422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:06.485275 16422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:52:06.485360 16422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:52:06.495679 16422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:35595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:06.497195 16422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:06.498605 16422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:06.525986 16429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:06.525885 16430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:06.525959 16432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:06.527170 16422 server_base.cc:1034] running on GCE node
I20260807 06:52:06.528187 16422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:06.530004 16422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:06.531273 16422 hybrid_clock.cc:648] HybridClock initialized: now 1786085526531225 us; error 69 us; skew 500 ppm
I20260807 06:52:06.531739 16422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:06 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085526, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:06.539245 16422 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:52:06.541329 16422 webserver.cc:460] Webserver started at http://127.27.54.126:44173/ using document root <none> and password file <none>
I20260807 06:52:06.542204 16422 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:06.542399 16422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:06.549427 16422 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260807 06:52:06.552834 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:06.554255 16422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:52:06.554489 16422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "d60b421756234074958a269dfd67caa3"
format_stamp: "Formatted at 2026-08-07 06:52:03 on dist-test-slave-3kk6"
server_key: "35ffc6af7f7ef15f0401ec0bfad5dce9"
server_key_iv: "c93717001ec8f54d0b5b9f1a9f9f548e"
server_key_version: "encryptionkey@0"
I20260807 06:52:06.555794 16422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:06.572167 16422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:06.576917 16422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:06.577256 16422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:06.599237 16422 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:35595
I20260807 06:52:06.599267 16508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:35595 every 8 connection(s)
I20260807 06:52:06.600988 16422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:52:06.608494 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 16422
I20260807 06:52:06.608561 16509 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:52:06.609407 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:43887
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:35595
--webserver_port=37335
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:52:06.612342 16509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap starting.
I20260807 06:52:06.633898 16509 log.cc:824] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:06.647055 16509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:52:06.647723 16509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap complete.
I20260807 06:52:06.657745 16509 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:06.658741 16509 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60b421756234074958a269dfd67caa3, State: Initialized, Role: FOLLOWER
I20260807 06:52:06.659442 16509 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:06.659719 16509 raft_consensus.cc:388] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:52:06.659883 16509 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:52:06.660108 16509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:52:06.664489 16509 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:06.665112 16509 leader_election.cc:302] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d60b421756234074958a269dfd67caa3; no voters:
I20260807 06:52:06.665823 16509 leader_election.cc:288] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:52:06.665978 16515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:52:06.668534 16509 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:52:06.671027 16515 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 LEADER]: Becoming Leader. State: Replica: d60b421756234074958a269dfd67caa3, State: Running, Role: LEADER
I20260807 06:52:06.671841 16515 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:06.675450 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60b421756234074958a269dfd67caa3. Latest consensus state: current_term: 2 leader_uuid: "d60b421756234074958a269dfd67caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } } }
I20260807 06:52:06.676023 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:06.675446 16516 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d60b421756234074958a269dfd67caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } } }
I20260807 06:52:06.676713 16516 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:06.680812 16527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:52:06.690117 16527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:52:06.691764 16527 catalog_manager.cc:1259] Loaded cluster ID: 2b2bb0249da24564b9ecb8abb0d80b84
I20260807 06:52:06.691977 16527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:52:06.696043 16527 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 06:52:06.714569 16527 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 06:52:06.725332 16527 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:52:06.727227 16527 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Loaded TSK: 0
I20260807 06:52:06.728801 16527 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:52:07.000893 16513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:07.001267 16513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:07.001447 16513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:07.009033 16513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:07.009219 16513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:07.009407 16513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:52:07.019726 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:43887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=37335
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:07.021174 16513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:07.022730 16513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:07.038128 16550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:07.038071 16549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:07.039309 16553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:07.039705 16513 server_base.cc:1034] running on GCE node
I20260807 06:52:07.040385 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:07.041453 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:07.042764 16513 hybrid_clock.cc:648] HybridClock initialized: now 1786085527042680 us; error 82 us; skew 500 ppm
I20260807 06:52:07.043115 16513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:07 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085527, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:07.046643 16513 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:52:07.048470 16513 webserver.cc:460] Webserver started at http://127.27.54.65:37335/ using document root <none> and password file <none>
I20260807 06:52:07.049182 16513 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:07.049365 16513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:07.055344 16513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 06:52:07.058324 16563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:07.059912 16513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:52:07.060108 16513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "5bf89563309345ba98ecdcb5d15aba99"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "538fcf99a25694ae710682107d9a6af3"
server_key_iv: "b09e1aa3a6c78a9d7f344fadf65c4211"
server_key_version: "encryptionkey@0"
I20260807 06:52:07.061210 16513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:07.076336 16513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:07.080689 16513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:07.080991 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:07.082188 16513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:07.083690 16513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:07.083827 16513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:07.083964 16513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:07.084055 16513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:07.120115 16513 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43887
I20260807 06:52:07.120234 16722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43887 every 8 connection(s)
I20260807 06:52:07.121703 16513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:52:07.123947 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 16513
I20260807 06:52:07.124845 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:44095
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:35595
--webserver_port=40649
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:52:07 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085527, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:07.149724 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:07.124782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:39795 (local address 127.27.54.126:35595)
0807 06:52:07.127235 (+ 2453us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:07.127262 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:07.127324 (+ 62us) server_negotiation.cc:407] Connection header received
0807 06:52:07.129055 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:07.129076 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:07.129589 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:07.130041 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:07.131694 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.133817 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.135563 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.136699 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.136913 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:07.141167 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:07.141175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:07.141191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:07.141232 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:07.144767 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:07.145606 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:07.145613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:07.145619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:07.145799 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:07.146288 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:07.146297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:07.146299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:07.146873 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:07.147097 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:07.147951 (+ 854us) server_negotiation.cc:299] Negotiation successful
0807 06:52:07.148378 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2195,"thread_start_us":2061,"threads_started":1}
I20260807 06:52:07.149727 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:07.125169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:39795)
0807 06:52:07.126293 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:07.126359 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:07.128381 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:07.130156 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:07.130164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:07.130831 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:07.131402 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:07.131420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.134030 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.134034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:07.135400 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:07.135413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.136954 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.136959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:07.137152 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:07.137947 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:07.137971 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:07.140840 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:07.144941 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:07.144951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:07.144968 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:07.145438 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:07.145927 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:07.145932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:07.145935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:07.146141 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:07.147024 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:07.147034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:07.147394 (+ 360us) client_negotiation.cc:769] Sending connection context
0807 06:52:07.147687 (+ 293us) client_negotiation.cc:240] Negotiation successful
0807 06:52:07.148304 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":400,"threads_started":1}
I20260807 06:52:07.152809 16723 heartbeater.cc:347] Connected to a master server at 127.27.54.126:35595
I20260807 06:52:07.153158 16723 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:07.154000 16723 heartbeater.cc:511] Master 127.27.54.126:35595 requested a full tablet report, sending...
I20260807 06:52:07.156270 16462 ts_manager.cc:194] Registered new tserver with Master: 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65:43887)
I20260807 06:52:07.157806 16462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:39795
W20260807 06:52:07.493633 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:07.494019 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:07.494208 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:07.502476 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:07.502748 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:07.502947 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:52:07.513322 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=40649
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:07.514897 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:07.516291 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:07.530076 16740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:07.530035 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:07.530103 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:07.531359 16731 server_base.cc:1034] running on GCE node
I20260807 06:52:07.532101 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:07.533128 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:07.534348 16731 hybrid_clock.cc:648] HybridClock initialized: now 1786085527534297 us; error 39 us; skew 500 ppm
I20260807 06:52:07.534739 16731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:07 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085527, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:07.538226 16731 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:52:07.539981 16731 webserver.cc:460] Webserver started at http://127.27.54.66:40649/ using document root <none> and password file <none>
I20260807 06:52:07.540704 16731 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:07.540889 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:07.547226 16731 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:52:07.550341 16751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:07.551939 16731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:52:07.552147 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "98bcf00a9ad8488784ab75c777a1a57c"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "88d9c6f372fe0cb01fc7d5a28d09508d"
server_key_iv: "8939ee45dd63d85a0ba124acdf1a7ae4"
server_key_version: "encryptionkey@0"
I20260807 06:52:07.553253 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:07.573681 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:07.578594 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:07.578999 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:07.580173 16731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:07.581807 16731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:07.581934 16731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:07.582084 16731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:07.582181 16731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:07.619421 16731 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:44095
I20260807 06:52:07.619484 16917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:44095 every 8 connection(s)
I20260807 06:52:07.620970 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:52:07.631250 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 16731
I20260807 06:52:07.632128 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:46351
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:35595
--webserver_port=45461
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:52:07 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085527, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:07.641886 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:07.623649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:39289 (local address 127.27.54.126:35595)
0807 06:52:07.623882 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:07.623889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:07.625190 (+ 1301us) server_negotiation.cc:407] Connection header received
0807 06:52:07.627100 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:07.627104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:07.627195 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:07.627304 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:07.628854 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.629560 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.631243 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.631780 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.631928 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:07.635749 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:07.635756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:07.635761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:07.635804 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:07.638500 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:07.639379 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:07.639385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:07.639396 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:07.639539 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:07.640026 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:07.640030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:07.640032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:07.640325 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:07.640488 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:07.641248 (+ 760us) server_negotiation.cc:299] Negotiation successful
0807 06:52:07.641512 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 06:52:07.643056 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:07.624251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:39289)
0807 06:52:07.624999 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:07.625040 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:07.626814 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:07.627529 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:07.627537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:07.628111 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:07.628673 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:07.628689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.629708 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.629712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:07.631069 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:07.631083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:07.632040 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:07.632048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:07.632322 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:07.633111 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:07.633139 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:07.635506 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:07.638698 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:07.638708 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:07.638727 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:07.639230 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:07.639664 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:07.639669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:07.639673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:07.639890 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:07.640462 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:07.640470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:07.640852 (+ 382us) client_negotiation.cc:769] Sending connection context
0807 06:52:07.641142 (+ 290us) client_negotiation.cc:240] Negotiation successful
0807 06:52:07.641715 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":293,"threads_started":1}
I20260807 06:52:07.645023 16918 heartbeater.cc:347] Connected to a master server at 127.27.54.126:35595
I20260807 06:52:07.645367 16918 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:07.646322 16918 heartbeater.cc:511] Master 127.27.54.126:35595 requested a full tablet report, sending...
I20260807 06:52:07.648160 16462 ts_manager.cc:194] Registered new tserver with Master: 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095)
I20260807 06:52:07.649035 16462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:39289
W20260807 06:52:08.018309 16923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:08.018625 16923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:08.018819 16923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:08.026422 16923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:08.026608 16923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:08.026811 16923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:52:08.037117 16923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:46351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=45461
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:08.038448 16923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:08.039803 16923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:08.053663 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:08.053691 16930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:08.053778 16929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:08.059294 16923 server_base.cc:1034] running on GCE node
I20260807 06:52:08.060086 16923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:08.061340 16923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:08.062623 16923 hybrid_clock.cc:648] HybridClock initialized: now 1786085528062574 us; error 53 us; skew 500 ppm
I20260807 06:52:08.063099 16923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:08 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085528, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:08.067142 16923 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:52:08.069108 16923 webserver.cc:460] Webserver started at http://127.27.54.67:45461/ using document root <none> and password file <none>
I20260807 06:52:08.070042 16923 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:08.070225 16923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:08.076742 16923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:52:08.079878 16941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:08.081550 16923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:52:08.081821 16923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "98bfe8095c6f4ba7be8f72954105f985"
format_stamp: "Formatted at 2026-08-07 06:52:05 on dist-test-slave-3kk6"
server_key: "8f958298132ac4baa82ac028ba2e825d"
server_key_iv: "302a2e68c50f4dc0ff55c5d5fff7c9a9"
server_key_version: "encryptionkey@0"
I20260807 06:52:08.083123 16923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:08.102267 16923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:08.107416 16923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:08.107749 16923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:08.108836 16923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:08.110318 16923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:08.110451 16923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:08.110599 16923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:08.110775 16923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 06:52:08.148932 16923 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:46351
I20260807 06:52:08.149051 17104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:46351 every 8 connection(s)
I20260807 06:52:08.150528 16923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:52:08.159538 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 16923
I20260807 06:52:08.161341 16723 heartbeater.cc:503] Master 127.27.54.126:35595 was elected leader, sending a full tablet report...
Aug 07 06:52:08 dist-test-slave-3kk6 krb5kdc[15664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085528, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:08.175753 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.153542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:38113 (local address 127.27.54.126:35595)
0807 06:52:08.154088 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.154093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.154990 (+ 897us) server_negotiation.cc:407] Connection header received
0807 06:52:08.157211 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.157218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.157321 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.157463 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:08.159313 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.160289 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.163259 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.163893 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.164085 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.168557 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:08.168562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:08.168566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:08.168600 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:08.172013 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.172845 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.172852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.172858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.173017 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.173584 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.173591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.173596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.174008 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:08.174194 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.175057 (+ 863us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.175327 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":317,"threads_started":1}
I20260807 06:52:08.176510 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.153948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:38113)
0807 06:52:08.154803 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:08.154852 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:08.156881 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:08.157758 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:08.157767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:08.158367 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:08.159109 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.159129 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.160505 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.160509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.163050 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.163068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.164155 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.164175 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.164446 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.165310 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:08.165336 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:08.168293 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:08.172193 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:08.172203 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:08.172230 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:08.172700 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:08.173147 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:08.173153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:08.173157 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:08.173423 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:08.174176 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:08.174183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:08.174537 (+ 354us) client_negotiation.cc:769] Sending connection context
0807 06:52:08.174891 (+ 354us) client_negotiation.cc:240] Negotiation successful
0807 06:52:08.175333 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":298,"threads_started":1}
I20260807 06:52:08.178121 17106 heartbeater.cc:347] Connected to a master server at 127.27.54.126:35595
I20260807 06:52:08.178467 17106 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:08.179338 17106 heartbeater.cc:511] Master 127.27.54.126:35595 requested a full tablet report, sending...
I20260807 06:52:08.180888 16462 ts_manager.cc:194] Registered new tserver with Master: 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67:46351)
I20260807 06:52:08.181815 16462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:38113
I20260807 06:52:08.182305 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.160842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.27.54.126:35595)
0807 06:52:08.167115 (+ 6273us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.167122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.167552 (+ 430us) server_negotiation.cc:407] Connection header received
0807 06:52:08.167771 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.167776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.167862 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.167984 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:08.169916 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.170935 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.172319 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.172748 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.172885 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.176292 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:08.176297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:08.176300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:08.176330 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:08.179313 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.180068 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.180076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.180080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.180215 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.180630 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.180635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.180639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.180937 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:08.181231 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.181603 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.181840 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6124,"thread_start_us":5964,"threads_started":1}
I20260807 06:52:08.184968 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:52:08.192103 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.185993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.27.54.126:35595)
0807 06:52:08.186172 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.186177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.186727 (+ 550us) server_negotiation.cc:407] Connection header received
0807 06:52:08.186990 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.186994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.187121 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.187253 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:52:08.187657 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.188441 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.189939 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.190353 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.190448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.191058 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:52:08.191565 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:52:08.191689 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.191836 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.191906 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:52:08.193128 16462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:52:08.200829 16462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:52:08.225664 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.214072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:46351 (local address 127.0.0.1:51484)
0807 06:52:08.215441 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:08.215496 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:08.215693 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:08.218161 (+ 2468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:08.218169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:08.218198 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:08.218486 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.218493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.220761 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.220767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.222692 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.222707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.224562 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.224569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.224800 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.224861 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 06:52:08.225019 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 06:52:08.225229 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":29,"thread_start_us":1798,"threads_started":1}
I20260807 06:52:08.225618 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.214530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.27.54.67:46351)
0807 06:52:08.217511 (+ 2981us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.217523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.217562 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:52:08.217659 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.217666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.217918 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.218171 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:08.218681 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.220600 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.222862 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.224324 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.224476 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.224722 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.225071 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.225246 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2783,"thread_start_us":2617,"threads_started":1}
I20260807 06:52:08.225891 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.213990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.27.54.65:43887)
0807 06:52:08.214694 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.214706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.215621 (+ 915us) server_negotiation.cc:407] Connection header received
0807 06:52:08.215960 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.215968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.216238 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.216412 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:08.217672 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.219555 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.222327 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.223942 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.224089 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.224361 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.225369 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.225553 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":735,"threads_started":1}
I20260807 06:52:08.226500 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.213663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43887 (local address 127.0.0.1:50468)
0807 06:52:08.215441 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:08.215490 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:08.215721 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:08.216793 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:08.216800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:08.216829 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:08.217395 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.217412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.219738 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.219743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.222119 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.222139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.224515 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.224522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.224834 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.224871 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:52:08.225008 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 06:52:08.225228 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"mutex_wait_us":68,"thread_start_us":1309,"threads_started":1}
I20260807 06:52:08.227262 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.214529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:44095 (local address 127.0.0.1:58292)
0807 06:52:08.216433 (+ 1904us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x61200001a79c. stack: 00007fa7eedf166e 00007fa7eedf1430 00007fa7ede785ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fa7eecbb8d8 00007fa7eecbd054 00007fa7f9cf3595 00007fa7f9cea11c 00007fa7eee14bc2 00000000009da5bb 00007fa7eee34fd7 00007fa7eee30bea 00007fa7eee35cee 00007fa7f58a4e4a
0807 06:52:08.217870 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:08.217891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:08.218042 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:08.219083 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:08.219088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:08.219112 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:08.219583 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.219596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.221444 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.221451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.224067 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.224082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.226280 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.226290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.226496 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.226541 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 06:52:08.226643 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 06:52:08.226822 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3208,"mutex_wait_us":24,"spinlock_wait_cycles":2477312,"thread_start_us":3060,"threads_started":1}
I20260807 06:52:08.228093 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.213618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.27.54.66:44095)
0807 06:52:08.214456 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.214467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.218011 (+ 3544us) server_negotiation.cc:407] Connection header received
0807 06:52:08.218744 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.218750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.218903 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.219104 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:08.219801 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.221211 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.224235 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.225762 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.225935 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.226131 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.227574 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.227758 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":1079,"threads_started":1}
I20260807 06:52:08.230041 16625 tablet_service.cc:1467] Processing CreateTablet for tablet 0c13cc0433f645669e209bf988bb89ab (DEFAULT_TABLE table=test-table [id=79f60289af814a19be9f74dd46568273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:08.230038 17010 tablet_service.cc:1467] Processing CreateTablet for tablet 0c13cc0433f645669e209bf988bb89ab (DEFAULT_TABLE table=test-table [id=79f60289af814a19be9f74dd46568273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:08.231380 17010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c13cc0433f645669e209bf988bb89ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:08.231496 16625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c13cc0433f645669e209bf988bb89ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:08.234829 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 0c13cc0433f645669e209bf988bb89ab (DEFAULT_TABLE table=test-table [id=79f60289af814a19be9f74dd46568273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:08.236312 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c13cc0433f645669e209bf988bb89ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:08.245138 17133 tablet_bootstrap.cc:492] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Bootstrap starting.
I20260807 06:52:08.248778 17134 tablet_bootstrap.cc:492] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Bootstrap starting.
I20260807 06:52:08.249840 17133 tablet_bootstrap.cc:654] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:08.251348 17133 log.cc:824] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:08.252916 17134 tablet_bootstrap.cc:654] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:08.254243 17134 log.cc:824] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:08.254739 17137 tablet_bootstrap.cc:492] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Bootstrap starting.
I20260807 06:52:08.255038 17133 tablet_bootstrap.cc:492] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: No bootstrap required, opened a new log
I20260807 06:52:08.255376 17133 ts_tablet_manager.cc:1397] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260807 06:52:08.257673 17134 tablet_bootstrap.cc:492] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: No bootstrap required, opened a new log
I20260807 06:52:08.257962 17134 ts_tablet_manager.cc:1397] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260807 06:52:08.259591 17137 tablet_bootstrap.cc:654] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:08.261011 17137 log.cc:824] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:08.264546 17137 tablet_bootstrap.cc:492] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: No bootstrap required, opened a new log
I20260807 06:52:08.264395 17133 raft_consensus.cc:348] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.264859 17137 ts_tablet_manager.cc:1397] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260807 06:52:08.264878 17133 raft_consensus.cc:374] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:08.264987 17133 raft_consensus.cc:729] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf89563309345ba98ecdcb5d15aba99, State: Initialized, Role: FOLLOWER
I20260807 06:52:08.265650 17133 consensus_queue.cc:260] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.268837 17134 raft_consensus.cc:348] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.269341 17134 raft_consensus.cc:374] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:08.269459 17134 raft_consensus.cc:729] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bfe8095c6f4ba7be8f72954105f985, State: Initialized, Role: FOLLOWER
I20260807 06:52:08.269891 17133 ts_tablet_manager.cc:1428] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260807 06:52:08.270103 17134 consensus_queue.cc:260] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.271797 17134 ts_tablet_manager.cc:1428] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260807 06:52:08.272217 17106 heartbeater.cc:503] Master 127.27.54.126:35595 was elected leader, sending a full tablet report...
I20260807 06:52:08.276675 17137 raft_consensus.cc:348] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.277109 17137 raft_consensus.cc:374] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:08.277230 17137 raft_consensus.cc:729] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bcf00a9ad8488784ab75c777a1a57c, State: Initialized, Role: FOLLOWER
I20260807 06:52:08.277837 17137 consensus_queue.cc:260] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.279050 16918 heartbeater.cc:503] Master 127.27.54.126:35595 was elected leader, sending a full tablet report...
I20260807 06:52:08.279834 17137 ts_tablet_manager.cc:1428] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260807 06:52:08.366562 17141 raft_consensus.cc:482] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:52:08.366984 17141 raft_consensus.cc:504] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.368777 17141 leader_election.cc:288] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095), 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67:46351)
W20260807 06:52:08.373353 16920 tablet.cc:2369] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:52:08.374204 16726 tablet.cc:2369] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:08.377509 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.369388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51263 (local address 127.27.54.67:46351)
0807 06:52:08.369584 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.369591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.370361 (+ 770us) server_negotiation.cc:407] Connection header received
0807 06:52:08.370610 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.370615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.370729 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.370823 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:08.371736 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.372904 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.375600 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.376424 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.376633 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.376730 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.377077 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.377222 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:52:08.377517 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.369284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:32829 (local address 127.27.54.66:44095)
0807 06:52:08.369506 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.369512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.370422 (+ 910us) server_negotiation.cc:407] Connection header received
0807 06:52:08.370604 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.370610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.370716 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.370842 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:08.371740 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.372884 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.375580 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.376386 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.376600 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.376717 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.377077 (+ 360us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.377219 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:52:08.377735 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.369299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:44095 (local address 127.27.54.65:32829)
0807 06:52:08.370224 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:08.370238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:08.370417 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:08.370883 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:08.370889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:08.370933 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:08.371546 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.371561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.373219 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.373225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.375376 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.375393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.376684 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.376696 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.376863 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.376927 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 06:52:08.377045 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:52:08.377211 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":115,"spinlock_wait_cycles":6016,"thread_start_us":534,"threads_started":1}
I20260807 06:52:08.377763 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.369118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:46351 (local address 127.27.54.65:51263)
0807 06:52:08.370200 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:08.370228 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:08.370460 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:08.370883 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:08.370889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:08.370928 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:08.371546 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.371561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.373026 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.373031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.375420 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:08.375434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.376718 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.376725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:08.376876 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.376934 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 06:52:08.377045 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:52:08.377222 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":160,"spinlock_wait_cycles":44160,"thread_start_us":655,"threads_started":1}
I20260807 06:52:08.378894 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c13cc0433f645669e209bf988bb89ab" candidate_uuid: "5bf89563309345ba98ecdcb5d15aba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bcf00a9ad8488784ab75c777a1a57c" is_pre_election: true
I20260807 06:52:08.378996 17037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c13cc0433f645669e209bf988bb89ab" candidate_uuid: "5bf89563309345ba98ecdcb5d15aba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bfe8095c6f4ba7be8f72954105f985" is_pre_election: true
I20260807 06:52:08.379444 16850 raft_consensus.cc:2454] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf89563309345ba98ecdcb5d15aba99 in term 0.
I20260807 06:52:08.379505 17037 raft_consensus.cc:2454] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf89563309345ba98ecdcb5d15aba99 in term 0.
I20260807 06:52:08.380244 16567 leader_election.cc:302] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5bf89563309345ba98ecdcb5d15aba99, 98bcf00a9ad8488784ab75c777a1a57c, 98bfe8095c6f4ba7be8f72954105f985; no voters:
I20260807 06:52:08.380635 17141 raft_consensus.cc:2781] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:52:08.380803 17141 raft_consensus.cc:482] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:52:08.380923 17141 raft_consensus.cc:3037] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:08.383453 17141 raft_consensus.cc:504] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.384073 17141 leader_election.cc:288] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [CANDIDATE]: Term 1 election: Requested vote from peers 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095), 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67:46351)
I20260807 06:52:08.384630 17037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c13cc0433f645669e209bf988bb89ab" candidate_uuid: "5bf89563309345ba98ecdcb5d15aba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bfe8095c6f4ba7be8f72954105f985"
I20260807 06:52:08.384615 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c13cc0433f645669e209bf988bb89ab" candidate_uuid: "5bf89563309345ba98ecdcb5d15aba99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98bcf00a9ad8488784ab75c777a1a57c"
I20260807 06:52:08.384874 16850 raft_consensus.cc:3037] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:08.384878 17037 raft_consensus.cc:3037] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:08.387482 16850 raft_consensus.cc:2454] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf89563309345ba98ecdcb5d15aba99 in term 1.
I20260807 06:52:08.387519 17037 raft_consensus.cc:2454] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf89563309345ba98ecdcb5d15aba99 in term 1.
I20260807 06:52:08.387979 16567 leader_election.cc:302] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf89563309345ba98ecdcb5d15aba99, 98bfe8095c6f4ba7be8f72954105f985; no voters:
I20260807 06:52:08.388335 17141 raft_consensus.cc:2781] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:08.388859 17141 raft_consensus.cc:686] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf89563309345ba98ecdcb5d15aba99, State: Running, Role: LEADER
I20260807 06:52:08.389446 17141 consensus_queue.cc:237] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } }
I20260807 06:52:08.394229 16462 catalog_manager.cc:5495] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65). New cstate: current_term: 1 leader_uuid: "5bf89563309345ba98ecdcb5d15aba99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bf89563309345ba98ecdcb5d15aba99" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 } health_report { overall_health: UNKNOWN } } }
W20260807 06:52:08.402559 17107 tablet.cc:2369] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:08.423641 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.418024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.27.54.65:43887)
0807 06:52:08.418296 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.418301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.418320 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:52:08.418535 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.418539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.418621 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.418819 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:52:08.419173 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.419899 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.421343 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.421782 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.421882 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.422409 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:52:08.423023 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:52:08.423137 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.423238 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.423322 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 06:52:08.432472 17037 raft_consensus.cc:1261] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Refusing update from remote peer 5bf89563309345ba98ecdcb5d15aba99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:08.432471 16850 raft_consensus.cc:1261] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Refusing update from remote peer 5bf89563309345ba98ecdcb5d15aba99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:08.433601 17147 consensus_queue.cc:1035] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bfe8095c6f4ba7be8f72954105f985" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 46351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:52:08.433995 17141 consensus_queue.cc:1035] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98bcf00a9ad8488784ab75c777a1a57c" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:52:08.446198 17151 mvcc.cc:205] Tried to move back new op lower bound from 7315806324448006144 to 7315806324287184896. Current Snapshot: MvccSnapshot[applied={T|T < 7315806324448006144}]
I20260807 06:52:08.463415 16461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:08.463730 16461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:08.468037 16461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Sending DeleteTablet for 3 replicas of tablet 0c13cc0433f645669e209bf988bb89ab
I20260807 06:52:08.469542 16625 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c13cc0433f645669e209bf988bb89ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:08 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:50468
I20260807 06:52:08.470465 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c13cc0433f645669e209bf988bb89ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:08 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:58292
I20260807 06:52:08.470705 17164 tablet_replica.cc:330] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: stopping tablet replica
I20260807 06:52:08.471205 17164 raft_consensus.cc:2229] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:52:08.471560 17165 tablet_replica.cc:330] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: stopping tablet replica
I20260807 06:52:08.471594 17010 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c13cc0433f645669e209bf988bb89ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:08 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:51484
I20260807 06:52:08.472015 17164 raft_consensus.cc:2258] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:08.472178 17165 raft_consensus.cc:2229] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:08.472748 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 16513
I20260807 06:52:08.472765 17165 raft_consensus.cc:2258] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:08.474735 17164 ts_tablet_manager.cc:1905] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:08.476303 17165 ts_tablet_manager.cc:1905] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:08.483083 17164 ts_tablet_manager.cc:1918] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:08.483330 17164 log.cc:1196] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/0c13cc0433f645669e209bf988bb89ab
I20260807 06:52:08.483942 17165 ts_tablet_manager.cc:1918] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:08.484016 17164 ts_tablet_manager.cc:1939] T 0c13cc0433f645669e209bf988bb89ab P 5bf89563309345ba98ecdcb5d15aba99: Deleting consensus metadata
I20260807 06:52:08.484251 17165 log.cc:1196] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/0c13cc0433f645669e209bf988bb89ab
I20260807 06:52:08.484926 17165 ts_tablet_manager.cc:1939] T 0c13cc0433f645669e209bf988bb89ab P 98bcf00a9ad8488784ab75c777a1a57c: Deleting consensus metadata
I20260807 06:52:08.486238 16446 catalog_manager.cc:4841] TS 5bf89563309345ba98ecdcb5d15aba99 (127.27.54.65:43887): tablet 0c13cc0433f645669e209bf988bb89ab (table test-table [id=79f60289af814a19be9f74dd46568273]) successfully deleted
I20260807 06:52:08.486986 16444 catalog_manager.cc:4841] TS 98bcf00a9ad8488784ab75c777a1a57c (127.27.54.66:44095): tablet 0c13cc0433f645669e209bf988bb89ab (table test-table [id=79f60289af814a19be9f74dd46568273]) successfully deleted
I20260807 06:52:08.491298 17166 tablet_replica.cc:330] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: stopping tablet replica
I20260807 06:52:08.491775 17166 raft_consensus.cc:2229] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:08.492340 17166 raft_consensus.cc:2258] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:08.494714 17166 ts_tablet_manager.cc:1905] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:08.495999 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.473772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.27.54.65:43887)
0807 06:52:08.473951 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.473958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.474135 (+ 177us) server_negotiation.cc:407] Connection header received
0807 06:52:08.474369 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.474374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.474467 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.474579 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:08.476755 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.478037 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.480484 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.481656 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.481803 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.487425 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:08.487439 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:08.487457 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:08.487494 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:08.492100 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.493285 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.493296 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.493300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.493494 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.494025 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.494032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.494037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.494542 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:08.494984 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.495333 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.495616 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:52:08.496881 16717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:08.502942 17166 ts_tablet_manager.cc:1918] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:08.503140 17166 log.cc:1196] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/0c13cc0433f645669e209bf988bb89ab
I20260807 06:52:08.503690 17166 ts_tablet_manager.cc:1939] T 0c13cc0433f645669e209bf988bb89ab P 98bfe8095c6f4ba7be8f72954105f985: Deleting consensus metadata
I20260807 06:52:08.505442 16445 catalog_manager.cc:4841] TS 98bfe8095c6f4ba7be8f72954105f985 (127.27.54.67:46351): tablet 0c13cc0433f645669e209bf988bb89ab (table test-table [id=79f60289af814a19be9f74dd46568273]) successfully deleted
I20260807 06:52:08.610863 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 16513
I20260807 06:52:08.642022 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 16731
I20260807 06:52:08.658239 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.642971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.27.54.66:44095)
0807 06:52:08.643220 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.643225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.643245 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:52:08.643378 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.643382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.643447 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.643536 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:08.645033 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.645818 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.647281 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.647681 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.647802 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.651056 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:08.651079 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:08.651096 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:08.651146 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:08.654757 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.655734 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.655743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.655745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.655898 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.656290 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.656294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.656296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.656818 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:08.657059 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.657530 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.657863 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:52:08.659010 16909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:08.751797 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 16731
I20260807 06:52:08.777639 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 16923
I20260807 06:52:08.793063 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:08.778583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.27.54.67:46351)
0807 06:52:08.778834 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:08.778839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:08.778862 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:52:08.778987 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:08.778991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:08.779065 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:08.779168 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:08.780736 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.781586 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.783170 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:08.783588 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:08.783741 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:08.786713 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:08.786735 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:08.786751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:08.786828 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:08.789901 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.790924 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.790932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.790934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.791086 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:08.791557 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:08.791562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:08.791564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:08.792026 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:08.792277 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:08.792579 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 06:52:08.792788 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260807 06:52:08.793802 17099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:08.887058 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 16923
I20260807 06:52:08.911226 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 16422
I20260807 06:52:08.912555 16503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:09.030064 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 16422
I20260807 06:52:09.061667 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:35595
--webserver_interface=127.27.54.126
--webserver_port=44173
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:09.420657 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:09.421000 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:09.421097 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:09.428591 17177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:52:09.428738 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:09.428787 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:09.428840 17177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:52:09.428886 17177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:52:09.439461 17177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:35595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:35595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:09.440905 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:09.442332 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:09.454932 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:09.454867 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:09.454898 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:09.456321 17177 server_base.cc:1034] running on GCE node
I20260807 06:52:09.457201 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:09.458822 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:09.460068 17177 hybrid_clock.cc:648] HybridClock initialized: now 1786085529460013 us; error 66 us; skew 500 ppm
I20260807 06:52:09.460485 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:09 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085529, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:09.464021 17177 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:52:09.465687 17177 webserver.cc:460] Webserver started at http://127.27.54.126:44173/ using document root <none> and password file <none>
I20260807 06:52:09.466439 17177 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:09.466598 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:09.472206 17177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:52:09.475235 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:09.476619 17177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:52:09.476842 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "d60b421756234074958a269dfd67caa3"
format_stamp: "Formatted at 2026-08-07 06:52:03 on dist-test-slave-3kk6"
server_key: "35ffc6af7f7ef15f0401ec0bfad5dce9"
server_key_iv: "c93717001ec8f54d0b5b9f1a9f9f548e"
server_key_version: "encryptionkey@0"
I20260807 06:52:09.477896 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:09.501860 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:09.506743 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:09.507062 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:09.527148 17177 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:35595
I20260807 06:52:09.527185 17266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:35595 every 8 connection(s)
I20260807 06:52:09.528780 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:52:09.534745 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 17177
I20260807 06:52:09.535663 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:43887
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:35595
--webserver_port=37335
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:52:09.536000 17269 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:52:09.539374 17269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap starting.
I20260807 06:52:09.575067 17269 log.cc:824] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:09.598587 17269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:52:09.599442 17269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3: Bootstrap complete.
I20260807 06:52:09.615290 17269 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:09.616878 17269 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60b421756234074958a269dfd67caa3, State: Initialized, Role: FOLLOWER
I20260807 06:52:09.617780 17269 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:09.618139 17269 raft_consensus.cc:388] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:52:09.618304 17269 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:52:09.618533 17269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:52:09.625277 17269 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:09.625954 17269 leader_election.cc:302] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d60b421756234074958a269dfd67caa3; no voters:
I20260807 06:52:09.626714 17269 leader_election.cc:288] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 06:52:09.626930 17274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 06:52:09.628360 17274 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [term 3 LEADER]: Becoming Leader. State: Replica: d60b421756234074958a269dfd67caa3, State: Running, Role: LEADER
I20260807 06:52:09.628994 17269 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:52:09.628914 17274 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } }
I20260807 06:52:09.631778 17277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60b421756234074958a269dfd67caa3. Latest consensus state: current_term: 3 leader_uuid: "d60b421756234074958a269dfd67caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } } }
I20260807 06:52:09.632217 17277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:09.632758 17276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d60b421756234074958a269dfd67caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60b421756234074958a269dfd67caa3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 35595 } } }
I20260807 06:52:09.633075 17276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60b421756234074958a269dfd67caa3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:09.644407 17288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:52:09.656971 17288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:52:09.659168 17288 catalog_manager.cc:1259] Loaded cluster ID: 2b2bb0249da24564b9ecb8abb0d80b84
I20260807 06:52:09.659327 17288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 06:52:09.662308 17288 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc75412e9ad google::LogMessage::Fail() at ??:0
@ 0x7fc75413188a google::LogMessage::SendToLog() at ??:0
@ 0x7fc75412e4de google::LogMessage::Flush() at ??:0
@ 0x7fc75412fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc769b61512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc769b5e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc769be6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc769be6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc756d7a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc756d7d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc756d7d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc756d58eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc75a0326db start_thread at ??:0
@ 0x7fc752fc861f clone at ??:0
W20260807 06:52:09.997130 17271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:09.997476 17271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:09.997581 17271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:10.005182 17271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:10.005332 17271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:10.005456 17271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:52:10.015717 17271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:43887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=37335
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:10.017138 17271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:10.018536 17271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:10.032756 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:10.032826 17333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:10.032830 17328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:10.034091 17271 server_base.cc:1034] running on GCE node
I20260807 06:52:10.034819 17271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:10.035825 17271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:10.037040 17271 hybrid_clock.cc:648] HybridClock initialized: now 1786085530036998 us; error 41 us; skew 500 ppm
I20260807 06:52:10.037400 17271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:10 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085530, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:10.040915 17271 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:52:10.042568 17271 webserver.cc:460] Webserver started at http://127.27.54.65:37335/ using document root <none> and password file <none>
I20260807 06:52:10.043412 17271 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:10.043563 17271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:10.049402 17271 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:52:10.052611 17341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:10.054116 17271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 06:52:10.054281 17271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "5bf89563309345ba98ecdcb5d15aba99"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "538fcf99a25694ae710682107d9a6af3"
server_key_iv: "b09e1aa3a6c78a9d7f344fadf65c4211"
server_key_version: "encryptionkey@0"
I20260807 06:52:10.055444 17271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:10.085245 17271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:10.089802 17271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:10.090078 17271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:10.091126 17271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:10.092572 17271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:10.092679 17271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:10.092765 17271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:10.092837 17271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:10.134533 17271 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43887
I20260807 06:52:10.134636 17498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43887 every 8 connection(s)
I20260807 06:52:10.136178 17271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:52:10.141546 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 17271
I20260807 06:52:10.142252 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:10.139682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:42361)
0807 06:52:10.140517 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:10.140940 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":337,"threads_started":1}
I20260807 06:52:10.142443 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:44095
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:35595
--webserver_port=40649
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:10.143195 17499 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:35595 (0 consecutive failures): Network error: Failed to ping master at 127.27.54.126:35595: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
I20260807 06:52:10.144354 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:10.143870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:51393)
0807 06:52:10.144074 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:10.144179 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 06:52:10.145376 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:10.144916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:45169)
0807 06:52:10.145099 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:10.145212 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 06:52:10.145639 17499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:52:10.515596 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:10.515992 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:10.516183 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:10.523787 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:10.523986 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:10.524194 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:52:10.534929 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=40649
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:10.536473 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:10.537935 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:10.552599 17511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:10.552670 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:10.552699 17510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:10.553743 17503 server_base.cc:1034] running on GCE node
I20260807 06:52:10.554567 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:10.555953 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:10.557199 17503 hybrid_clock.cc:648] HybridClock initialized: now 1786085530557121 us; error 70 us; skew 500 ppm
I20260807 06:52:10.557590 17503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:10 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085530, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:10.561645 17503 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:52:10.563419 17503 webserver.cc:460] Webserver started at http://127.27.54.66:40649/ using document root <none> and password file <none>
I20260807 06:52:10.564260 17503 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:10.564431 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:10.570448 17503 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 06:52:10.573616 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:10.575109 17503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:52:10.575311 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "98bcf00a9ad8488784ab75c777a1a57c"
format_stamp: "Formatted at 2026-08-07 06:52:04 on dist-test-slave-3kk6"
server_key: "88d9c6f372fe0cb01fc7d5a28d09508d"
server_key_iv: "8939ee45dd63d85a0ba124acdf1a7ae4"
server_key_version: "encryptionkey@0"
I20260807 06:52:10.576432 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:10.602222 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:10.606765 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:10.607098 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:10.608220 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:10.609683 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:10.609819 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:10.609966 17503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:10.610064 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:10.651252 17503 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:44095
I20260807 06:52:10.651352 17672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:44095 every 8 connection(s)
I20260807 06:52:10.652880 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:52:10.657706 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 17503
I20260807 06:52:10.658073 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:10.656099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:49125)
0807 06:52:10.656846 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:10.657102 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":295,"threads_started":1}
I20260807 06:52:10.658419 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:46351
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:35595
--webserver_port=45461
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:10.658983 17674 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:35595 (0 consecutive failures): Network error: Failed to ping master at 127.27.54.126:35595: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
I20260807 06:52:10.660023 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:10.659614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:56061)
0807 06:52:10.659783 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:10.659884 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 06:52:10.660986 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:10.660632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:55615)
0807 06:52:10.660782 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:10.660875 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 06:52:10.661258 17674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 06:52:11.012560 17679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:11.012854 17679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:11.013019 17679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:11.020295 17679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:11.020409 17679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:11.020584 17679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:52:11.030885 17679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:46351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=45461
--tserver_master_addrs=127.27.54.126:35595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:11.032330 17679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:11.033756 17679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:11.047228 17688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:11.047202 17689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:11.047140 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:11.048485 17679 server_base.cc:1034] running on GCE node
I20260807 06:52:11.049110 17679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:11.050114 17679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:11.051343 17679 hybrid_clock.cc:648] HybridClock initialized: now 1786085531051307 us; error 38 us; skew 500 ppm
I20260807 06:52:11.051669 17679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:11 dist-test-slave-3kk6 krb5kdc[15664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085531, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:11.054967 17679 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:52:11.056536 17679 webserver.cc:460] Webserver started at http://127.27.54.67:45461/ using document root <none> and password file <none>
I20260807 06:52:11.057224 17679 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:11.057379 17679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:11.063011 17679 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 06:52:11.065750 17700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:11.067286 17679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 06:52:11.067473 17679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "98bfe8095c6f4ba7be8f72954105f985"
format_stamp: "Formatted at 2026-08-07 06:52:05 on dist-test-slave-3kk6"
server_key: "8f958298132ac4baa82ac028ba2e825d"
server_key_iv: "302a2e68c50f4dc0ff55c5d5fff7c9a9"
server_key_version: "encryptionkey@0"
I20260807 06:52:11.068538 17679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:11.089740 17679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:11.094703 17679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:11.095122 17679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:11.096244 17679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:11.097723 17679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:11.097857 17679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:11.098006 17679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:11.098099 17679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:11.133745 17679 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:46351
I20260807 06:52:11.133874 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:46351 every 8 connection(s)
I20260807 06:52:11.135396 17679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:52:11.140854 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:11.138952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:58627)
0807 06:52:11.139786 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:11.140007 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":350,"threads_started":1}
W20260807 06:52:11.141693 17861 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:35595 (0 consecutive failures): Network error: Failed to ping master at 127.27.54.126:35595: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
I20260807 06:52:11.142308 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 17679
I20260807 06:52:11.142702 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:11.142265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:57147)
0807 06:52:11.142459 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:11.142549 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 06:52:11.144330 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:11.143857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:51859)
0807 06:52:11.144045 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:11.144152 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 06:52:11.144655 17861 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:52:11.147047 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:11.146166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:49171)
0807 06:52:11.146639 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:11.146818 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":225,"threads_started":1}
I20260807 06:52:11.662918 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:11.661936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:34711)
0807 06:52:11.662467 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:11.662573 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":263,"threads_started":1}
I20260807 06:52:12.146214 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:12.145254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:47169)
0807 06:52:12.145848 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:12.145975 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260807 06:52:12.148806 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:12.147973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:44745)
0807 06:52:12.148539 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:12.148630 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":326,"threads_started":1}
I20260807 06:52:12.664690 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:12.663850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:55929)
0807 06:52:12.664427 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:12.664516 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":311,"threads_started":1}
I20260807 06:52:13.148126 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:13.147329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:37819)
0807 06:52:13.147852 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:13.147950 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":269,"threads_started":1}
I20260807 06:52:13.150494 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:13.149700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:37119)
0807 06:52:13.150169 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:13.150256 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
I20260807 06:52:13.666566 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:13.665662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:33695)
0807 06:52:13.666175 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:13.666320 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":284,"threads_started":1}
I20260807 06:52:14.150741 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:14.149289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:32913)
0807 06:52:14.150382 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:14.150472 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":854,"threads_started":1}
I20260807 06:52:14.152212 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:14.151438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:54111)
0807 06:52:14.151972 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:14.152051 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
I20260807 06:52:14.668577 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:14.667782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:47447)
0807 06:52:14.668267 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:14.668371 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":248,"threads_started":1}
I20260807 06:52:15.152549 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:15.151749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:59117)
0807 06:52:15.152276 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:15.152373 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":277,"threads_started":1}
I20260807 06:52:15.153980 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:15.153183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:44083)
0807 06:52:15.153660 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:15.153768 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260807 06:52:15.670423 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:15.669623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:60173)
0807 06:52:15.670144 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:15.670238 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
I20260807 06:52:16.154347 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:16.153580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:56963)
0807 06:52:16.154085 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:16.154168 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":280,"threads_started":1}
I20260807 06:52:16.155768 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:16.154922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:37367)
0807 06:52:16.155486 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:16.155587 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":327,"threads_started":1}
I20260807 06:52:16.672459 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:16.671448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:55397)
0807 06:52:16.672134 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:16.672258 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":316,"threads_started":1}
I20260807 06:52:17.156101 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:17.155318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:41873)
0807 06:52:17.155823 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:17.155928 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":253,"threads_started":1}
I20260807 06:52:17.157720 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:17.156825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:42605)
0807 06:52:17.157427 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:17.157519 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":297,"threads_started":1}
I20260807 06:52:17.674324 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:17.673419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:49891)
0807 06:52:17.674003 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:17.674094 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":269,"threads_started":1}
I20260807 06:52:18.158192 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:18.157218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:46111)
0807 06:52:18.157819 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:18.157919 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":244,"threads_started":1}
I20260807 06:52:18.159564 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:18.158796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:35867)
0807 06:52:18.159333 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:18.159415 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":272,"threads_started":1}
I20260807 06:52:18.676291 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:18.675462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:35225)
0807 06:52:18.676014 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:18.676112 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":241,"threads_started":1}
I20260807 06:52:19.160353 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:19.159378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:54271)
0807 06:52:19.160012 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:19.160112 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":281,"threads_started":1}
I20260807 06:52:19.161528 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:19.160555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:50899)
0807 06:52:19.161244 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:19.161368 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":318,"threads_started":1}
I20260807 06:52:19.678233 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:19.677367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:46849)
0807 06:52:19.677899 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:19.677996 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":274,"threads_started":1}
I20260807 06:52:20.162593 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:20.161575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:53265)
0807 06:52:20.162245 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:20.162351 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":310,"threads_started":1}
I20260807 06:52:20.163389 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:20.162522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:50065)
0807 06:52:20.163095 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:20.163195 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":369,"threads_started":1}
I20260807 06:52:20.680358 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:20.679460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:33189)
0807 06:52:20.680070 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:20.680181 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":375,"threads_started":1}
I20260807 06:52:21.164674 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:21.163643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:56253)
0807 06:52:21.164283 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:21.164399 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":378,"threads_started":1}
I20260807 06:52:21.165346 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:21.164335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:54965)
0807 06:52:21.164968 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:21.165085 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":276,"threads_started":1}
I20260807 06:52:21.682439 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:21.681459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:47763)
0807 06:52:21.682004 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:21.682155 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":248,"threads_started":1}
I20260807 06:52:22.166810 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:22.165733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:60925)
0807 06:52:22.166435 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:22.166529 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":316,"threads_started":1}
I20260807 06:52:22.167125 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:22.166286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:51703)
0807 06:52:22.166834 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:22.166947 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
I20260807 06:52:22.684230 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:22.683474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:44739)
0807 06:52:22.683951 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:22.684041 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":268,"threads_started":1}
I20260807 06:52:23.168803 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:23.167851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:32803)
0807 06:52:23.168472 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:23.168591 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
I20260807 06:52:23.168955 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:23.168042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:34743)
0807 06:52:23.168619 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:23.168733 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":298,"threads_started":1}
I20260807 06:52:23.686236 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:23.685321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:58873)
0807 06:52:23.685887 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:23.686016 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":315,"threads_started":1}
I20260807 06:52:24.170905 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:24.169974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:51671)
0807 06:52:24.170497 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:24.170605 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260807 06:52:24.171329 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:24.169974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:46443)
0807 06:52:24.170970 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:24.171100 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":721,"threads_started":1}
I20260807 06:52:24.688454 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:24.687495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:33671)
0807 06:52:24.688092 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:24.688241 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":304,"threads_started":1}
I20260807 06:52:25.173135 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:25.172316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:48093)
0807 06:52:25.172863 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:25.172952 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":309,"threads_started":1}
I20260807 06:52:25.174062 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:25.172286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:43021)
0807 06:52:25.173763 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:25.173881 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":1239,"threads_started":1}
I20260807 06:52:25.690753 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:25.689750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.66:49191)
0807 06:52:25.690329 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:25.690495 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":332,"threads_started":1}
I20260807 06:52:26.156239 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 17271
I20260807 06:52:26.175675 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:26.174751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.67:52285)
0807 06:52:26.175301 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:26.175429 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":298,"threads_started":1}
I20260807 06:52:26.176422 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:26.174956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:35595 (local address 127.27.54.65:41671)
0807 06:52:26.175695 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:26.176048 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":416,"threads_started":1}
I20260807 06:52:26.182027 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:26.157749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.27.54.65:43887)
0807 06:52:26.158311 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:26.158326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:26.158366 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:52:26.160128 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:26.160137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:26.160780 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:26.161261 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:26.162834 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:26.164126 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:26.165441 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:26.166129 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:26.166295 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:26.174857 (+ 8562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:26.174870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:26.174895 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:26.174953 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:26.178588 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:26.179501 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:26.179509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:26.179515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:26.179684 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:26.180081 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:26.180084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:26.180086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:26.180618 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:26.180899 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:26.181316 (+ 417us) server_negotiation.cc:299] Negotiation successful
0807 06:52:26.181749 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":257,"threads_started":1}
I20260807 06:52:26.183518 17493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:26.270493 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 17271
I20260807 06:52:26.295828 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 17503
I20260807 06:52:26.314898 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:26.296777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.27.54.66:44095)
0807 06:52:26.297356 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:26.297370 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:26.297408 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:52:26.299165 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:26.299174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:26.299602 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:26.299961 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:26.301579 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:26.302971 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:26.304246 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:26.304917 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:26.305147 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:26.308236 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:26.308244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:26.308261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:26.308306 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:26.311338 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:26.312109 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:26.312117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:26.312124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:26.312305 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:26.312716 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:26.312724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:26.312728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:26.313272 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:26.313608 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:26.314062 (+ 454us) server_negotiation.cc:299] Negotiation successful
0807 06:52:26.314517 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":285,"threads_started":1}
I20260807 06:52:26.316319 17668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:26.403982 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 17503
I20260807 06:52:26.429849 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 17679
I20260807 06:52:26.449060 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:26.430919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.27.54.67:46351)
0807 06:52:26.431531 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:26.431546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:26.431590 (+ 44us) server_negotiation.cc:407] Connection header received
0807 06:52:26.433289 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:26.433299 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:26.433731 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:26.434113 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:26.435902 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:26.437234 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:26.438573 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:26.439341 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:26.439567 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:26.442359 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:26.442368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:26.442384 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:26.442423 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:26.445455 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:26.446166 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:26.446173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:26.446180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:26.446357 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:26.446837 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:26.446842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:26.446848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:26.447396 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:26.447758 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:26.448280 (+ 522us) server_negotiation.cc:299] Negotiation successful
0807 06:52:26.448722 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
I20260807 06:52:26.450742 17855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:26.539186 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 17679
2026-08-07T06:52:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25324 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:52:26 dist-test-slave-3kk6 krb5kdc[17948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:52:26 dist-test-slave-3kk6 krb5kdc[17948](info): set up 2 sockets
Aug 07 06:52:26 dist-test-slave-3kk6 krb5kdc[17948](info): commencing operation
krb5kdc: starting...
W20260807 06:52:27.884536 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:52:28 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:52:28.242794 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40693
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:28.603763 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:28.604149 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:28.604328 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:28.611701 17963 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 06:52:28.611860 17963 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 06:52:28.611933 17963 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 06:52:28.612015 17963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:52:28.612126 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:28.612251 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:28.612341 17963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:52:28.612424 17963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:52:28.622522 17963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:28.623993 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:28.625451 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:28.638236 17972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:28.638638 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:28.638231 17973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:28.638820 17963 server_base.cc:1034] running on GCE node
I20260807 06:52:28.639966 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:28.641484 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:28.642776 17963 hybrid_clock.cc:648] HybridClock initialized: now 1786085548642687 us; error 61 us; skew 500 ppm
I20260807 06:52:28.643136 17963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:28 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085548, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:28.646589 17963 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:52:28.648154 17963 webserver.cc:460] Webserver started at http://127.27.54.126:44583/ using document root <none> and password file <none>
I20260807 06:52:28.648897 17963 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:28.649036 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:28.649389 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:28.652627 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "e0b8626855814334835047958d3033e2"
format_stamp: "Formatted at 2026-08-07 06:52:28 on dist-test-slave-3kk6"
server_key: "25bafb3be6057fd5304100b0a9f99631"
server_key_iv: "42e0cc069f885ac9e093fe1b50ce57ec"
server_key_version: "encryptionkey@0"
I20260807 06:52:28.653497 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "e0b8626855814334835047958d3033e2"
format_stamp: "Formatted at 2026-08-07 06:52:28 on dist-test-slave-3kk6"
server_key: "25bafb3be6057fd5304100b0a9f99631"
server_key_iv: "42e0cc069f885ac9e093fe1b50ce57ec"
server_key_version: "encryptionkey@0"
I20260807 06:52:28.659135 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:52:28.662724 17987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:28.664074 17963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:52:28.664263 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "e0b8626855814334835047958d3033e2"
format_stamp: "Formatted at 2026-08-07 06:52:28 on dist-test-slave-3kk6"
server_key: "25bafb3be6057fd5304100b0a9f99631"
server_key_iv: "42e0cc069f885ac9e093fe1b50ce57ec"
server_key_version: "encryptionkey@0"
I20260807 06:52:28.664505 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:28.679307 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:28.683790 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:28.684114 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:28.704072 17963 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40693
I20260807 06:52:28.704098 18061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40693 every 8 connection(s)
I20260807 06:52:28.705574 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:52:28.706810 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 17963
I20260807 06:52:28.707068 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:52:28.707959 27865 external_mini_cluster.cc:1428] Setting key 0f90d111cc2f55ff1a6b2a9a83d3bc1b
I20260807 06:52:28.710187 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:52:28 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:28.725090 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap starting.
I20260807 06:52:28.729518 18062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:28.731048 18062 log.cc:824] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:28.734885 18062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: No bootstrap required, opened a new log
I20260807 06:52:28.736663 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:28.710193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.27.54.126:40693)
0807 06:52:28.711135 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:28.711151 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:28.711206 (+ 55us) server_negotiation.cc:407] Connection header received
0807 06:52:28.713578 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:28.713606 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:28.714165 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:28.714672 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:28.716273 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:28.718462 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:28.720068 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:28.721210 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:28.721408 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:28.726349 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:28.726361 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:28.726383 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:28.726449 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:28.730976 (+ 4527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:28.732078 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:28.732088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:28.732095 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:28.732278 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:28.732774 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:28.732782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:28.732789 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:28.733502 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:28.733831 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:28.734525 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 06:52:28.735218 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":422,"threads_started":1}
I20260807 06:52:28.743781 18062 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:28.744174 18062 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:28.744287 18062 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b8626855814334835047958d3033e2, State: Initialized, Role: FOLLOWER
I20260807 06:52:28.744988 18062 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:28.745250 18062 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:52:28.745414 18062 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:52:28.745609 18062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:28.748136 18062 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:28.748751 18062 leader_election.cc:302] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b8626855814334835047958d3033e2; no voters:
I20260807 06:52:28.749454 18062 leader_election.cc:288] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:52:28.749562 18068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:28.750401 18068 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 LEADER]: Becoming Leader. State: Replica: e0b8626855814334835047958d3033e2, State: Running, Role: LEADER
I20260807 06:52:28.751022 18068 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:28.751724 18062 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:52:28.754390 18070 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0b8626855814334835047958d3033e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } } }
I20260807 06:52:28.754292 18071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b8626855814334835047958d3033e2. Latest consensus state: current_term: 1 leader_uuid: "e0b8626855814334835047958d3033e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } } }
I20260807 06:52:28.754739 18071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:28.754755 18070 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:28.755743 18078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:52:28.764437 18078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:52:28.774231 18078 catalog_manager.cc:1347] Generated new cluster ID: 97fb94ab0de942319c7482163757aca7
I20260807 06:52:28.774385 18078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:52:28.787588 18078 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:52:28.789037 18078 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:52:28.796941 18078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Generated new TSK 0
I20260807 06:52:28.797852 18078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:52:28.883074 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40693
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:29.260291 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:29.260603 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:29.260730 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:29.268378 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:29.268549 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:29.268759 18096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:52:29.279239 18096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:29.280730 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:29.282167 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:29.296203 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:29.296331 18107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:29.296564 18106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:29.298017 18096 server_base.cc:1034] running on GCE node
I20260807 06:52:29.298636 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:29.299697 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:29.300915 18096 hybrid_clock.cc:648] HybridClock initialized: now 1786085549300866 us; error 35 us; skew 500 ppm
I20260807 06:52:29.301256 18096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:29 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:29.304831 18096 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:52:29.306281 18096 webserver.cc:460] Webserver started at http://127.27.54.65:45565/ using document root <none> and password file <none>
I20260807 06:52:29.307070 18096 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:29.307178 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:29.307464 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:29.310472 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "d61d795500871f56d373ec8efbd9910b"
server_key_iv: "f81fe1b78abca24788d53960f779b05e"
server_key_version: "encryptionkey@0"
I20260807 06:52:29.311324 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "d61d795500871f56d373ec8efbd9910b"
server_key_iv: "f81fe1b78abca24788d53960f779b05e"
server_key_version: "encryptionkey@0"
I20260807 06:52:29.316809 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 06:52:29.320393 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:29.321771 18096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:52:29.321966 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "d61d795500871f56d373ec8efbd9910b"
server_key_iv: "f81fe1b78abca24788d53960f779b05e"
server_key_version: "encryptionkey@0"
I20260807 06:52:29.322225 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:29.334985 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:29.339648 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:29.339990 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:29.341027 18096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:29.342422 18096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:29.342499 18096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:29.342690 18096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:29.342762 18096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 06:52:29.376933 18096 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43423
I20260807 06:52:29.376986 18281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43423 every 8 connection(s)
I20260807 06:52:29.378423 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:52:29.383586 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 18096
I20260807 06:52:29.383854 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:52:29.384577 27865 external_mini_cluster.cc:1428] Setting key fc37537f2aad357cf959c6a4d1f3bb21
Aug 07 06:52:29 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:29.401716 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:29.381292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:52313 (local address 127.27.54.126:40693)
0807 06:52:29.381801 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:29.381806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:29.382847 (+ 1041us) server_negotiation.cc:407] Connection header received
0807 06:52:29.384986 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:29.384992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:29.385103 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:29.385234 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:29.387190 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.387919 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.389601 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.390011 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.390154 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:29.394565 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:29.394571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:29.394574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:29.394613 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:29.397583 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:29.398758 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:29.398762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:29.398765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:29.398909 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:29.399482 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:29.399486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:29.399488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:29.399765 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:29.399900 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:29.400951 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0807 06:52:29.401240 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260807 06:52:29.402556 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:29.381809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:52313)
0807 06:52:29.382586 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:29.382632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:29.384666 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:29.385570 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:29.385582 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:29.386196 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:29.386999 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:29.387021 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.388110 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.388114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:29.389420 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:29.389434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.390286 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.390293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:29.390549 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:29.391389 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:29.391413 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:29.394204 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:29.397782 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:29.397796 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:29.397818 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:29.398558 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:29.399033 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:29.399040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:29.399046 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:29.399354 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:29.399908 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:29.399919 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:29.400463 (+ 544us) client_negotiation.cc:769] Sending connection context
0807 06:52:29.400769 (+ 306us) client_negotiation.cc:240] Negotiation successful
0807 06:52:29.401214 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":297,"threads_started":1}
I20260807 06:52:29.404417 18282 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40693
I20260807 06:52:29.404860 18282 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:29.405663 18282 heartbeater.cc:511] Master 127.27.54.126:40693 requested a full tablet report, sending...
I20260807 06:52:29.407934 18013 ts_manager.cc:194] Registered new tserver with Master: 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423)
I20260807 06:52:29.409454 18013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:52313
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:52:29.458357 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40693
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:29.824925 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:29.825309 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:29.825495 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:29.833184 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:29.833384 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:29.833572 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:52:29.844156 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:29.845641 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:29.847200 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:29.862025 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:29.861991 18303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:29.862062 18301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:29.863406 18293 server_base.cc:1034] running on GCE node
I20260807 06:52:29.864059 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:29.865077 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:29.866294 18293 hybrid_clock.cc:648] HybridClock initialized: now 1786085549866223 us; error 63 us; skew 500 ppm
I20260807 06:52:29.866803 18293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:29 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:29.870332 18293 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:52:29.872118 18293 webserver.cc:460] Webserver started at http://127.27.54.66:46089/ using document root <none> and password file <none>
I20260807 06:52:29.872864 18293 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:29.873008 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:29.873358 18293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:29.876606 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "561a8cc9cc9f46d69dde810280185324"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "789d4ce10597e42964d440c7fd2d2393"
server_key_iv: "0914709928d587212e9e6cae489da3dc"
server_key_version: "encryptionkey@0"
I20260807 06:52:29.877506 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "561a8cc9cc9f46d69dde810280185324"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "789d4ce10597e42964d440c7fd2d2393"
server_key_iv: "0914709928d587212e9e6cae489da3dc"
server_key_version: "encryptionkey@0"
I20260807 06:52:29.883348 18293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260807 06:52:29.887063 18312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:29.888592 18293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:52:29.888783 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "561a8cc9cc9f46d69dde810280185324"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "789d4ce10597e42964d440c7fd2d2393"
server_key_iv: "0914709928d587212e9e6cae489da3dc"
server_key_version: "encryptionkey@0"
I20260807 06:52:29.889015 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:29.915288 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:29.919749 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:29.920077 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:29.921185 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:29.922735 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:29.922823 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:29.922988 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:29.923043 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:29.959849 18293 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:39487
I20260807 06:52:29.959951 18478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:39487 every 8 connection(s)
I20260807 06:52:29.961347 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:52:29.963671 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 18293
I20260807 06:52:29.963884 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:52:29.964409 27865 external_mini_cluster.cc:1428] Setting key 52b766cb2fbdce034efe6aedd70709b9
Aug 07 06:52:29 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085549, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:29.987258 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:29.964445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:49081 (local address 127.27.54.126:40693)
0807 06:52:29.964986 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:29.964993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:29.966066 (+ 1073us) server_negotiation.cc:407] Connection header received
0807 06:52:29.968933 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:29.968938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:29.969069 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:29.969175 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:29.971606 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.972425 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.975133 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.975621 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.975752 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:29.980792 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:29.980798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:29.980801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:29.980842 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:29.983750 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:29.984650 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:29.984655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:29.984657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:29.984799 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:29.985301 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:29.985306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:29.985308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:29.985605 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:29.985777 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:29.986548 (+ 771us) server_negotiation.cc:299] Negotiation successful
0807 06:52:29.986851 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":267,"threads_started":1}
I20260807 06:52:29.988364 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:29.965070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:49081)
0807 06:52:29.965895 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:29.965943 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:29.968587 (+ 2644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:29.969512 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:29.969530 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:29.970436 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:29.971391 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:29.971422 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.972609 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.972618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:29.974912 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:29.974935 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:29.975911 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:29.975921 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:29.976292 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:29.977155 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:29.977191 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:29.980454 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:29.983949 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:29.983961 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:29.983982 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:29.984490 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:29.984936 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:29.984943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:29.984946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:29.985163 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:29.985756 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:29.985765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:29.986147 (+ 382us) client_negotiation.cc:769] Sending connection context
0807 06:52:29.986443 (+ 296us) client_negotiation.cc:240] Negotiation successful
0807 06:52:29.987068 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":325,"threads_started":1}
I20260807 06:52:29.990160 18479 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40693
I20260807 06:52:29.990511 18479 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:29.991372 18479 heartbeater.cc:511] Master 127.27.54.126:40693 requested a full tablet report, sending...
I20260807 06:52:29.993119 18013 ts_manager.cc:194] Registered new tserver with Master: 561a8cc9cc9f46d69dde810280185324 (127.27.54.66:39487)
I20260807 06:52:29.993992 18013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:49081
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:52:30.042402 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40693
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:52:30.412736 18282 heartbeater.cc:503] Master 127.27.54.126:40693 was elected leader, sending a full tablet report...
W20260807 06:52:30.413591 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:30.413882 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:30.414052 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:30.421831 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:30.421970 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:30.422151 18489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:52:30.432642 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:30.434108 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:30.435590 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:30.449590 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:30.449735 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:30.449637 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:30.450616 18489 server_base.cc:1034] running on GCE node
I20260807 06:52:30.451349 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:30.452543 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:30.453794 18489 hybrid_clock.cc:648] HybridClock initialized: now 1786085550453762 us; error 88 us; skew 500 ppm
I20260807 06:52:30.454142 18489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:30 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:30.457875 18489 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:52:30.459542 18489 webserver.cc:460] Webserver started at http://127.27.54.67:35183/ using document root <none> and password file <none>
I20260807 06:52:30.460295 18489 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:30.460464 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:30.460810 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:52:30.464092 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "35a9b32904d84367a843b3c31550d1c8"
format_stamp: "Formatted at 2026-08-07 06:52:30 on dist-test-slave-3kk6"
server_key: "8e7b46e8ef5b7adc6e750a7c6f1c6d0c"
server_key_iv: "1fbd447a0bba2bd25f9d54bc9052a15d"
server_key_version: "encryptionkey@0"
I20260807 06:52:30.464922 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "35a9b32904d84367a843b3c31550d1c8"
format_stamp: "Formatted at 2026-08-07 06:52:30 on dist-test-slave-3kk6"
server_key: "8e7b46e8ef5b7adc6e750a7c6f1c6d0c"
server_key_iv: "1fbd447a0bba2bd25f9d54bc9052a15d"
server_key_version: "encryptionkey@0"
I20260807 06:52:30.470923 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 06:52:30.474834 18507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:30.476290 18489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:52:30.476500 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "35a9b32904d84367a843b3c31550d1c8"
format_stamp: "Formatted at 2026-08-07 06:52:30 on dist-test-slave-3kk6"
server_key: "8e7b46e8ef5b7adc6e750a7c6f1c6d0c"
server_key_iv: "1fbd447a0bba2bd25f9d54bc9052a15d"
server_key_version: "encryptionkey@0"
I20260807 06:52:30.476738 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:30.502013 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:30.507275 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:30.507637 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:30.508766 18489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:30.510324 18489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:30.510435 18489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:30.510605 18489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:30.510699 18489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 06:52:30.548116 18489 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:38967
I20260807 06:52:30.548214 18665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:38967 every 8 connection(s)
I20260807 06:52:30.549526 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:52:30.558107 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 18489
I20260807 06:52:30.558337 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:52:30.558926 27865 external_mini_cluster.cc:1428] Setting key a4516cc2c57150f6445f205645364726
Aug 07 06:52:30 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085550, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:30.571595 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.552460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:36015 (local address 127.27.54.126:40693)
0807 06:52:30.552956 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.552961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.554100 (+ 1139us) server_negotiation.cc:407] Connection header received
0807 06:52:30.556239 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.556246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.556354 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.556482 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:30.558227 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.559228 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.561205 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.561640 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.561785 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.565450 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:30.565455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:30.565459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:30.565497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:30.568518 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:30.569351 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:30.569355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:30.569357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:30.569483 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:30.569937 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:30.569940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:30.569943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:30.570217 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:30.570346 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.571124 (+ 778us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.571306 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
I20260807 06:52:30.572639 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.553077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:36015)
0807 06:52:30.553906 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:30.553947 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:30.555869 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:30.556747 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:30.556757 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:30.557436 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:30.558034 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.558053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.559406 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.559410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.561059 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.561072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.561903 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.561911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.562153 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.562910 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:30.562933 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:30.565214 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:30.568672 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:30.568679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:30.568694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:30.569244 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:30.569587 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:30.569590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:30.569592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:30.569832 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:30.570356 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:30.570362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:30.570764 (+ 402us) client_negotiation.cc:769] Sending connection context
0807 06:52:30.571027 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 06:52:30.571449 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":339,"threads_started":1}
I20260807 06:52:30.574141 18666 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40693
I20260807 06:52:30.574489 18666 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:30.575307 18666 heartbeater.cc:511] Master 127.27.54.126:40693 requested a full tablet report, sending...
I20260807 06:52:30.576782 18013 ts_manager.cc:194] Registered new tserver with Master: 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67:38967)
I20260807 06:52:30.577574 18013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:36015
I20260807 06:52:30.587575 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:52:30.606344 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.592636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.27.54.126:40693)
0807 06:52:30.592848 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.592853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.593516 (+ 663us) server_negotiation.cc:407] Connection header received
0807 06:52:30.593665 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.593669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.593740 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.593900 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:30.595526 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.596287 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.597625 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.598026 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.598124 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.600743 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:30.600748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:30.600752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:30.600788 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:30.603666 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:30.604422 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:30.604430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:30.604436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:30.604583 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:30.604958 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:30.604962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:30.604964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:30.605297 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:30.605485 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.605773 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.605963 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:52:30.610738 18013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:52:30.617851 18013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:52:30.656808 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.630700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:39487 (local address 127.0.0.1:44986)
0807 06:52:30.645780 (+ 15080us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:30.645862 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:30.646176 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:30.647571 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:30.647583 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:30.647630 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:30.648070 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.648087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.650583 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.650588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.653473 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.653491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.655756 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.655764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.655957 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.655989 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:52:30.656175 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 06:52:30.656321 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14772,"mutex_wait_us":2,"thread_start_us":15729,"threads_started":1}
I20260807 06:52:30.657608 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.631267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43423 (local address 127.0.0.1:44336)
0807 06:52:30.646199 (+ 14932us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:30.646220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:30.646332 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:30.648620 (+ 2288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:30.648624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:30.648651 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:30.649076 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.649092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.651132 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.651139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.653512 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.653522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.655470 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.655477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.655704 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.655763 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 06:52:30.655918 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:52:30.656229 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14800,"mutex_wait_us":1,"thread_start_us":14678,"threads_started":1}
I20260807 06:52:30.657891 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.631083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.27.54.66:39487)
0807 06:52:30.645501 (+ 14418us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.645512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.646490 (+ 978us) server_negotiation.cc:407] Connection header received
0807 06:52:30.646614 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.646622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.646950 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.647180 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:30.648327 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.650414 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.653669 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.655228 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.655384 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.655600 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.657403 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.657587 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14181,"thread_start_us":14439,"threads_started":1}
I20260807 06:52:30.657092 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.631610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.27.54.65:43423)
0807 06:52:30.648004 (+ 16394us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.648016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.648052 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:52:30.648152 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.648159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.648391 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.648630 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:30.649311 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.650954 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.653707 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.654898 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.655047 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.655317 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.656466 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.656689 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16148,"thread_start_us":15936,"threads_started":1}
I20260807 06:52:30.662127 18375 tablet_service.cc:1467] Processing CreateTablet for tablet dc22244477c145feb2e0f240d180f7d8 (DEFAULT_TABLE table=test-table [id=4b788e08dee74b37aadd7c8ae64f37c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:30.663705 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc22244477c145feb2e0f240d180f7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:30.662127 18188 tablet_service.cc:1467] Processing CreateTablet for tablet dc22244477c145feb2e0f240d180f7d8 (DEFAULT_TABLE table=test-table [id=4b788e08dee74b37aadd7c8ae64f37c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:30.665807 18188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc22244477c145feb2e0f240d180f7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:30.668927 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.648368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:38967 (local address 127.0.0.1:55326)
0807 06:52:30.649401 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:30.649415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:30.649564 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:30.650613 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:30.650620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:30.650682 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:30.651309 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.651326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.653217 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.653223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.656867 (+ 3644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.656885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.668101 (+ 11216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.668107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.668264 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.668329 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 06:52:30.668465 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 06:52:30.668656 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":773,"threads_started":1}
I20260807 06:52:30.670193 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.647545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.27.54.67:38967)
0807 06:52:30.648380 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.648390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.649528 (+ 1138us) server_negotiation.cc:407] Connection header received
0807 06:52:30.649793 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.649802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.650043 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.650250 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:30.651613 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.653063 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.657089 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.658639 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.658806 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.669493 (+ 10687us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.669595 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.669819 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":462,"threads_started":1}
I20260807 06:52:30.674449 18576 tablet_service.cc:1467] Processing CreateTablet for tablet dc22244477c145feb2e0f240d180f7d8 (DEFAULT_TABLE table=test-table [id=4b788e08dee74b37aadd7c8ae64f37c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:30.677708 18576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc22244477c145feb2e0f240d180f7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:30.677985 18693 tablet_bootstrap.cc:492] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Bootstrap starting.
I20260807 06:52:30.681769 18693 tablet_bootstrap.cc:654] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:30.682286 18694 tablet_bootstrap.cc:492] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Bootstrap starting.
I20260807 06:52:30.683501 18693 log.cc:824] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:30.686476 18694 tablet_bootstrap.cc:654] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:30.687994 18694 log.cc:824] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:30.688290 18693 tablet_bootstrap.cc:492] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: No bootstrap required, opened a new log
I20260807 06:52:30.688632 18693 ts_tablet_manager.cc:1397] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260807 06:52:30.691689 18694 tablet_bootstrap.cc:492] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: No bootstrap required, opened a new log
I20260807 06:52:30.692006 18694 ts_tablet_manager.cc:1397] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 06:52:30.694440 18698 tablet_bootstrap.cc:492] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Bootstrap starting.
I20260807 06:52:30.697710 18698 tablet_bootstrap.cc:654] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:30.698928 18698 log.cc:824] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:30.700343 18693 raft_consensus.cc:348] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.700871 18693 raft_consensus.cc:374] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:30.700978 18693 raft_consensus.cc:729] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561a8cc9cc9f46d69dde810280185324, State: Initialized, Role: FOLLOWER
I20260807 06:52:30.701689 18693 consensus_queue.cc:260] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.702147 18698 tablet_bootstrap.cc:492] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: No bootstrap required, opened a new log
I20260807 06:52:30.702441 18698 ts_tablet_manager.cc:1397] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 06:52:30.703315 18694 raft_consensus.cc:348] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.703918 18694 raft_consensus.cc:374] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:30.704032 18694 raft_consensus.cc:729] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4072d3d4f2df42b7bfab3c24cf6eed85, State: Initialized, Role: FOLLOWER
I20260807 06:52:30.704766 18694 consensus_queue.cc:260] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.710840 18693 ts_tablet_manager.cc:1428] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Time spent starting tablet: real 0.022s user 0.016s sys 0.007s
I20260807 06:52:30.711378 18479 heartbeater.cc:503] Master 127.27.54.126:40693 was elected leader, sending a full tablet report...
W20260807 06:52:30.713552 18481 tablet.cc:2369] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:30.716257 18698 raft_consensus.cc:348] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.716754 18698 raft_consensus.cc:374] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:30.716868 18698 raft_consensus.cc:729] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a9b32904d84367a843b3c31550d1c8, State: Initialized, Role: FOLLOWER
I20260807 06:52:30.718397 18698 consensus_queue.cc:260] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.723053 18666 heartbeater.cc:503] Master 127.27.54.126:40693 was elected leader, sending a full tablet report...
I20260807 06:52:30.723502 18694 ts_tablet_manager.cc:1428] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Time spent starting tablet: real 0.031s user 0.020s sys 0.012s
I20260807 06:52:30.725584 18698 ts_tablet_manager.cc:1428] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
W20260807 06:52:30.801493 18667 tablet.cc:2369] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:30.811033 18704 raft_consensus.cc:482] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:52:30.811388 18704 raft_consensus.cc:504] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.813170 18704 leader_election.cc:288] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 561a8cc9cc9f46d69dde810280185324 (127.27.54.66:39487), 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423)
I20260807 06:52:30.819075 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.813515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:39487 (local address 127.27.54.67:40515)
0807 06:52:30.813769 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:30.813793 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:30.813931 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:30.814441 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:30.814446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:30.814475 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:30.814760 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.814771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.815810 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.815815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.817110 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.817124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.818263 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.818271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.818478 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.818503 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:52:30.818618 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 06:52:30.818782 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 06:52:30.819077 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.813743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:40515 (local address 127.27.54.66:39487)
0807 06:52:30.814151 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.814156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.814170 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:52:30.814231 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.814235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.814292 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.814404 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:30.814918 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.815645 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.817285 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.818100 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.818281 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.818360 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.818674 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.818804 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260807 06:52:30.820223 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.813723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:57797 (local address 127.27.54.65:43423)
0807 06:52:30.813948 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.813955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.814973 (+ 1018us) server_negotiation.cc:407] Connection header received
0807 06:52:30.815176 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.815180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.815239 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.815339 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:30.815919 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.816628 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.818279 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.819139 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.819339 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.819402 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.819949 (+ 547us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.820049 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:52:30.820438 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.814072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43423 (local address 127.27.54.67:57797)
0807 06:52:30.814859 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:30.814874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:30.815033 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:30.815360 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:30.815364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:30.815391 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:30.815748 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.815762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.816807 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.816811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.818110 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:30.818122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.819666 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.819670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:30.819793 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.819815 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:52:30.819927 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 06:52:30.820073 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":400,"threads_started":1}
I20260807 06:52:30.820541 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc22244477c145feb2e0f240d180f7d8" candidate_uuid: "35a9b32904d84367a843b3c31550d1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561a8cc9cc9f46d69dde810280185324" is_pre_election: true
I20260807 06:52:30.821179 18404 raft_consensus.cc:2454] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a9b32904d84367a843b3c31550d1c8 in term 0.
I20260807 06:52:30.821451 18217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc22244477c145feb2e0f240d180f7d8" candidate_uuid: "35a9b32904d84367a843b3c31550d1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" is_pre_election: true
I20260807 06:52:30.821924 18509 leader_election.cc:302] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a9b32904d84367a843b3c31550d1c8, 561a8cc9cc9f46d69dde810280185324; no voters:
I20260807 06:52:30.822105 18217 raft_consensus.cc:2454] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a9b32904d84367a843b3c31550d1c8 in term 0.
I20260807 06:52:30.822364 18704 raft_consensus.cc:2781] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:52:30.822541 18704 raft_consensus.cc:482] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:52:30.822706 18704 raft_consensus.cc:3037] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:30.825073 18704 raft_consensus.cc:504] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.825968 18704 leader_election.cc:288] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [CANDIDATE]: Term 1 election: Requested vote from peers 561a8cc9cc9f46d69dde810280185324 (127.27.54.66:39487), 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423)
I20260807 06:52:30.826499 18404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc22244477c145feb2e0f240d180f7d8" candidate_uuid: "35a9b32904d84367a843b3c31550d1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "561a8cc9cc9f46d69dde810280185324"
I20260807 06:52:30.826581 18217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc22244477c145feb2e0f240d180f7d8" candidate_uuid: "35a9b32904d84367a843b3c31550d1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
I20260807 06:52:30.826784 18404 raft_consensus.cc:3037] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:30.826889 18217 raft_consensus.cc:3037] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:30.829249 18217 raft_consensus.cc:2454] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a9b32904d84367a843b3c31550d1c8 in term 1.
I20260807 06:52:30.829252 18404 raft_consensus.cc:2454] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a9b32904d84367a843b3c31550d1c8 in term 1.
I20260807 06:52:30.829730 18510 leader_election.cc:302] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a9b32904d84367a843b3c31550d1c8, 4072d3d4f2df42b7bfab3c24cf6eed85; no voters:
I20260807 06:52:30.830102 18704 raft_consensus.cc:2781] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:30.830605 18704 raft_consensus.cc:686] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 1 LEADER]: Becoming Leader. State: Replica: 35a9b32904d84367a843b3c31550d1c8, State: Running, Role: LEADER
I20260807 06:52:30.831214 18704 consensus_queue.cc:237] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } }
I20260807 06:52:30.836090 18013 catalog_manager.cc:5495] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67). New cstate: current_term: 1 leader_uuid: "35a9b32904d84367a843b3c31550d1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20260807 06:52:30.860751 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.855056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.27.54.67:38967)
0807 06:52:30.855322 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.855327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.855348 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:52:30.855551 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.855555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.855652 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.855785 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:52:30.856162 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.857004 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.858529 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.858992 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.859093 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.859608 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:52:30.860132 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:52:30.860250 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.860369 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.860439 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 06:52:30.869653 18404 raft_consensus.cc:1261] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Refusing update from remote peer 35a9b32904d84367a843b3c31550d1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:30.869678 18217 raft_consensus.cc:1261] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Refusing update from remote peer 35a9b32904d84367a843b3c31550d1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:30.870738 18704 consensus_queue.cc:1035] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:52:30.871054 18707 consensus_queue.cc:1035] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:52:30.882010 18284 tablet.cc:2369] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:30.896754 18013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:30.897063 18013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:30.901516 18013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Sending DeleteTablet for 3 replicas of tablet dc22244477c145feb2e0f240d180f7d8
I20260807 06:52:30.902899 18576 tablet_service.cc:1514] Processing DeleteTablet for tablet dc22244477c145feb2e0f240d180f7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:30 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:55326
I20260807 06:52:30.902904 18375 tablet_service.cc:1514] Processing DeleteTablet for tablet dc22244477c145feb2e0f240d180f7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:30 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:44986
I20260807 06:52:30.903694 18188 tablet_service.cc:1514] Processing DeleteTablet for tablet dc22244477c145feb2e0f240d180f7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:30 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:44336
I20260807 06:52:30.903900 18723 tablet_replica.cc:330] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: stopping tablet replica
I20260807 06:52:30.904384 18723 raft_consensus.cc:2229] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:30.904695 18726 tablet_replica.cc:330] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: stopping tablet replica
I20260807 06:52:30.904947 18723 raft_consensus.cc:2258] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:30.905009 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 18096
I20260807 06:52:30.905512 18726 raft_consensus.cc:2229] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:30.905854 18724 tablet_replica.cc:330] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: stopping tablet replica
I20260807 06:52:30.906147 18726 raft_consensus.cc:2258] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:30.906276 18724 raft_consensus.cc:2229] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:52:30.907917 18724 raft_consensus.cc:2258] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:30.908670 18726 ts_tablet_manager.cc:1905] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:30.910519 18724 ts_tablet_manager.cc:1905] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:30.912822 18723 ts_tablet_manager.cc:1905] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:30.917372 18726 ts_tablet_manager.cc:1918] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:30.917629 18726 log.cc:1196] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/dc22244477c145feb2e0f240d180f7d8
I20260807 06:52:30.918288 18726 ts_tablet_manager.cc:1939] T dc22244477c145feb2e0f240d180f7d8 P 4072d3d4f2df42b7bfab3c24cf6eed85: Deleting consensus metadata
I20260807 06:52:30.920475 17991 catalog_manager.cc:4841] TS 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423): tablet dc22244477c145feb2e0f240d180f7d8 (table test-table [id=4b788e08dee74b37aadd7c8ae64f37c1]) successfully deleted
I20260807 06:52:30.920879 18723 ts_tablet_manager.cc:1918] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:30.921046 18723 log.cc:1196] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/dc22244477c145feb2e0f240d180f7d8
I20260807 06:52:30.921590 18723 ts_tablet_manager.cc:1939] T dc22244477c145feb2e0f240d180f7d8 P 561a8cc9cc9f46d69dde810280185324: Deleting consensus metadata
I20260807 06:52:30.923468 17990 catalog_manager.cc:4841] TS 561a8cc9cc9f46d69dde810280185324 (127.27.54.66:39487): tablet dc22244477c145feb2e0f240d180f7d8 (table test-table [id=4b788e08dee74b37aadd7c8ae64f37c1]) successfully deleted
I20260807 06:52:30.925987 18724 ts_tablet_manager.cc:1918] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:30.926177 18724 log.cc:1196] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/dc22244477c145feb2e0f240d180f7d8
I20260807 06:52:30.926961 18724 ts_tablet_manager.cc:1939] T dc22244477c145feb2e0f240d180f7d8 P 35a9b32904d84367a843b3c31550d1c8: Deleting consensus metadata
I20260807 06:52:30.928481 17990 catalog_manager.cc:4841] TS 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67:38967): tablet dc22244477c145feb2e0f240d180f7d8 (table test-table [id=4b788e08dee74b37aadd7c8ae64f37c1]) successfully deleted
Aug 07 06:52:30 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:52:30.942359 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:30.906427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.27.54.65:43423)
0807 06:52:30.906611 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:30.906618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:30.917625 (+ 11007us) server_negotiation.cc:407] Connection header received
0807 06:52:30.917947 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:30.917953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:30.918057 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:30.918184 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:30.925193 (+ 7009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.926397 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.928651 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:30.929114 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:30.929216 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:30.934804 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:30.934826 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:30.934847 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:30.934896 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:30.939003 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:30.939784 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:30.939795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:30.939799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:30.939985 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:30.940537 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:30.940543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:30.940546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:30.941050 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:30.941395 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:30.941688 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 06:52:30.941932 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 06:52:30.943282 18271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:31.051409 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 18096
I20260807 06:52:31.075326 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 18293
Aug 07 06:52:31 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:52:31.093366 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:31.076305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.27.54.66:39487)
0807 06:52:31.076524 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:31.076530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:31.076552 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:52:31.076663 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:31.076669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:31.076755 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:31.076882 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:31.078449 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:31.079330 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:31.080792 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:31.081218 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:31.081373 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:31.086330 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:31.086347 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:31.086362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:31.086400 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:31.089692 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:31.090908 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:31.090915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:31.090917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:31.091057 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:31.091510 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:31.091513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:31.091515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:31.091950 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:31.092191 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:31.092834 (+ 643us) server_negotiation.cc:299] Negotiation successful
0807 06:52:31.093027 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 06:52:31.094338 18469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:31.186913 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 18293
I20260807 06:52:31.219066 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 18489
Aug 07 06:52:31 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:52:31.241099 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:31.219944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.27.54.67:38967)
0807 06:52:31.220152 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:31.220157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:31.220237 (+ 80us) server_negotiation.cc:407] Connection header received
0807 06:52:31.220458 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:31.220466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:31.220543 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:31.220635 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:31.222720 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:31.223598 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:31.225609 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:31.226088 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:31.226336 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:31.234244 (+ 7908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:31.234259 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:31.234275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:31.234323 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:31.237740 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:31.238590 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:31.238601 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:31.238603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:31.238832 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:31.239277 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:31.239291 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:31.239296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:31.239696 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:31.240051 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:31.240439 (+ 388us) server_negotiation.cc:299] Negotiation successful
0807 06:52:31.240709 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 06:52:31.241834 18661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:31.371326 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 18489
I20260807 06:52:31.403857 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 17963
I20260807 06:52:31.405067 18055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:31.518106 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 17963
I20260807 06:52:51.547765 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40693
--webserver_interface=127.27.54.126
--webserver_port=44583
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:52:51.926996 18735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:51.927392 18735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:51.927572 18735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:51.935235 18735 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 06:52:51.935402 18735 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 06:52:51.935499 18735 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 06:52:51.935596 18735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:52:51.935698 18735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:51.935832 18735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:51.935920 18735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:52:51.936004 18735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:52:51.946588 18735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:51.948107 18735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:51.949584 18735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:51.977926 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:51.977902 18750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:51.977855 18748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:51.979046 18735 server_base.cc:1034] running on GCE node
I20260807 06:52:51.980029 18735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:51.981709 18735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:51.982981 18735 hybrid_clock.cc:648] HybridClock initialized: now 1786085571982914 us; error 58 us; skew 500 ppm
I20260807 06:52:51.983402 18735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:51 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085571, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:51.987617 18735 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:52:51.989634 18735 webserver.cc:460] Webserver started at http://127.27.54.126:44583/ using document root <none> and password file <none>
I20260807 06:52:51.990528 18735 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:51.990729 18735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:51.997507 18735 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 06:52:52.001048 18761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:52.002759 18735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:52:52.003001 18735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "e0b8626855814334835047958d3033e2"
format_stamp: "Formatted at 2026-08-07 06:52:28 on dist-test-slave-3kk6"
server_key: "25bafb3be6057fd5304100b0a9f99631"
server_key_iv: "42e0cc069f885ac9e093fe1b50ce57ec"
server_key_version: "encryptionkey@0"
I20260807 06:52:52.004204 18735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:52.027428 18735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:52.032496 18735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:52.032876 18735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:52.054869 18735 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40693
I20260807 06:52:52.054898 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40693 every 8 connection(s)
I20260807 06:52:52.056682 18735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:52:52.057407 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 18735
I20260807 06:52:52.058624 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:43423
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:40693
--webserver_port=45565
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:52:52.064831 18834 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:52:52.068668 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap starting.
I20260807 06:52:52.090796 18834 log.cc:824] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:52.108144 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:52:52.109107 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap complete.
I20260807 06:52:52.121889 18834 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:52.122970 18834 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b8626855814334835047958d3033e2, State: Initialized, Role: FOLLOWER
I20260807 06:52:52.123744 18834 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:52.124015 18834 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:52:52.124150 18834 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:52:52.124351 18834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 1 FOLLOWER]: Advancing to term 2
I20260807 06:52:52.128610 18834 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:52.129208 18834 leader_election.cc:302] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b8626855814334835047958d3033e2; no voters:
I20260807 06:52:52.129935 18834 leader_election.cc:288] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 06:52:52.130102 18838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 06:52:52.131542 18838 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 LEADER]: Becoming Leader. State: Replica: e0b8626855814334835047958d3033e2, State: Running, Role: LEADER
I20260807 06:52:52.132230 18838 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:52:52.132702 18834 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:52:52.135548 18841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b8626855814334835047958d3033e2. Latest consensus state: current_term: 2 leader_uuid: "e0b8626855814334835047958d3033e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } } }
I20260807 06:52:52.143287 18841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:52.135578 18839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0b8626855814334835047958d3033e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } } }
I20260807 06:52:52.148737 18839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: This master's current role is: LEADER
I20260807 06:52:52.152017 18848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:52:52.165036 18848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:52:52.167198 18848 catalog_manager.cc:1259] Loaded cluster ID: 97fb94ab0de942319c7482163757aca7
I20260807 06:52:52.167424 18848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:52:52.172689 18848 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:52:52.174683 18848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Loaded TSK: 0
I20260807 06:52:52.196708 18861 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:52:52.210376 18848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Generated new TSK 1
I20260807 06:52:52.210906 18848 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:52:52.496551 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:52.496968 18836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:52.497162 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:52.505990 18836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:52.506206 18836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:52.506402 18836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:52:52.518827 18836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:43423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=45565
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:52.520467 18836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:52.522068 18836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:52.539702 18872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:52.539702 18869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:52.540021 18867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:52.540759 18836 server_base.cc:1034] running on GCE node
I20260807 06:52:52.541495 18836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:52.542568 18836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:52.543905 18836 hybrid_clock.cc:648] HybridClock initialized: now 1786085572543836 us; error 65 us; skew 500 ppm
I20260807 06:52:52.544296 18836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:52 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:52.548125 18836 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:52:52.549923 18836 webserver.cc:460] Webserver started at http://127.27.54.65:45565/ using document root <none> and password file <none>
I20260807 06:52:52.550765 18836 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:52.550951 18836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:52.557528 18836 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 06:52:52.560894 18880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:52.562494 18836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:52:52.562721 18836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "d61d795500871f56d373ec8efbd9910b"
server_key_iv: "f81fe1b78abca24788d53960f779b05e"
server_key_version: "encryptionkey@0"
I20260807 06:52:52.563868 18836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:52.587749 18836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:52.592952 18836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:52.593282 18836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:52.594448 18836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:52.596002 18836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:52.596097 18836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:52.596266 18836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:52.596369 18836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:52.633774 18836 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43423
I20260807 06:52:52.634044 19047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43423 every 8 connection(s)
I20260807 06:52:52.635535 18836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:52:52.638528 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 18836
I20260807 06:52:52.639472 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:39487
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:40693
--webserver_port=46089
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:52:52 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:52.664583 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:52.638804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:45415 (local address 127.27.54.126:40693)
0807 06:52:52.639574 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:52.639595 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:52.640473 (+ 878us) server_negotiation.cc:407] Connection header received
0807 06:52:52.642906 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:52.642928 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:52.643421 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:52.643853 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:52.645942 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:52.648149 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:52.650070 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:52.651306 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:52.651557 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:52.656030 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:52.656040 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:52.656057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:52.656101 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:52.659405 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:52.660342 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:52.660356 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:52.660362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:52.660546 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:52.661037 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:52.661042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:52.661046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:52.661598 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:52.661819 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:52.662720 (+ 901us) server_negotiation.cc:299] Negotiation successful
0807 06:52:52.663170 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":300,"threads_started":1}
I20260807 06:52:52.664598 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:52.638980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:45415)
0807 06:52:52.640249 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:52.640301 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:52.642176 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:52.644101 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:52.644113 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:52.645022 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:52.645654 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:52.645672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:52.648368 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:52.648372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:52.649898 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:52.649913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:52.651674 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:52.651684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:52.652035 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:52.652840 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:52.652870 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:52.655717 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:52.659630 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:52.659645 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:52.659659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:52.660168 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:52.660691 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:52.660696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:52.660704 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:52.660914 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:52.661760 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:52.661769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:52.662136 (+ 367us) client_negotiation.cc:769] Sending connection context
0807 06:52:52.662451 (+ 315us) client_negotiation.cc:240] Negotiation successful
0807 06:52:52.663121 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":728,"threads_started":1}
I20260807 06:52:52.667907 19049 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40693
I20260807 06:52:52.668336 19049 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:52.669488 19049 heartbeater.cc:511] Master 127.27.54.126:40693 requested a full tablet report, sending...
I20260807 06:52:52.672130 18787 ts_manager.cc:194] Registered new tserver with Master: 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423)
I20260807 06:52:52.673732 18787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:45415
W20260807 06:52:53.099069 19056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:53.099399 19056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:53.099593 19056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:53.107678 19056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:53.107844 19056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:53.108052 19056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:52:53.118440 19056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:39487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=46089
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:53.119932 19056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:53.121382 19056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:53.135490 19062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:53.135594 19063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:53.135594 19065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:53.136682 19056 server_base.cc:1034] running on GCE node
I20260807 06:52:53.137377 19056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:53.138435 19056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:53.139647 19056 hybrid_clock.cc:648] HybridClock initialized: now 1786085573139581 us; error 64 us; skew 500 ppm
I20260807 06:52:53.139999 19056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:53 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085573, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:53.148351 19056 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:52:53.150152 19056 webserver.cc:460] Webserver started at http://127.27.54.66:46089/ using document root <none> and password file <none>
I20260807 06:52:53.151022 19056 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:53.151201 19056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:53.157215 19056 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 06:52:53.160198 19074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:53.161755 19056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:52:53.161935 19056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "561a8cc9cc9f46d69dde810280185324"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "789d4ce10597e42964d440c7fd2d2393"
server_key_iv: "0914709928d587212e9e6cae489da3dc"
server_key_version: "encryptionkey@0"
I20260807 06:52:53.163096 19056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:53.197328 19056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:53.203073 19056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:53.203392 19056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:53.204526 19056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:53.206053 19056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:53.206156 19056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:53.206301 19056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:53.206399 19056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:53.242136 19056 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:39487
I20260807 06:52:53.242238 19253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:39487 every 8 connection(s)
I20260807 06:52:53.243901 19056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:52:53.250059 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 19056
I20260807 06:52:53.250939 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:38967
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:40693
--webserver_port=35183
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 06:52:53 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085573, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:53.267041 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.246963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:56565 (local address 127.27.54.126:40693)
0807 06:52:53.247433 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.247438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.248525 (+ 1087us) server_negotiation.cc:407] Connection header received
0807 06:52:53.250606 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.250612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.250749 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.250900 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:53.252657 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.253486 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.255302 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.255744 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.255895 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.260255 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:53.260261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:53.260264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:53.260303 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:53.263421 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.264316 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.264321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.264324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.264475 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.265027 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.265032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.265034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.265350 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:53.265520 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.266327 (+ 807us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.266599 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260807 06:52:53.268406 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.247587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:56565)
0807 06:52:53.248338 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.248387 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.250259 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.251222 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.251240 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.251835 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:53.252460 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.252480 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.253700 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.253705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.255145 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.255159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.256021 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.256029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.256287 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.257229 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:53.257256 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:53.259956 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:53.263611 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:53.263623 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:53.263656 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:53.264154 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:53.264602 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:53.264607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:53.264611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:53.264881 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:53.265494 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:53.265504 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:53.265921 (+ 417us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.266218 (+ 297us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.266936 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":295,"threads_started":1}
I20260807 06:52:53.270262 19254 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40693
I20260807 06:52:53.270612 19254 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:53.271428 19254 heartbeater.cc:511] Master 127.27.54.126:40693 requested a full tablet report, sending...
I20260807 06:52:53.273124 18787 ts_manager.cc:194] Registered new tserver with Master: 561a8cc9cc9f46d69dde810280185324 (127.27.54.66:39487)
I20260807 06:52:53.273993 18787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:56565
W20260807 06:52:53.626745 19259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:52:53.627142 19259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:52:53.627360 19259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:52:53.635178 19259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:52:53.635313 19259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:52:53.635488 19259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:52:53.645807 19259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:38967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=35183
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:52:53.647379 19259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:52:53.648757 19259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:52:53.663792 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:53.663877 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:52:53.663831 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:52:53.664794 19259 server_base.cc:1034] running on GCE node
I20260807 06:52:53.665473 19259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:52:53.666556 19259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:52:53.667824 19259 hybrid_clock.cc:648] HybridClock initialized: now 1786085573667764 us; error 59 us; skew 500 ppm
I20260807 06:52:53.668215 19259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:52:53 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085573, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:52:53.671890 19259 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:52:53.673483 19259 webserver.cc:460] Webserver started at http://127.27.54.67:35183/ using document root <none> and password file <none>
I20260807 06:52:53.674189 19259 fs_manager.cc:359] Metadata directory not provided
I20260807 06:52:53.674317 19259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:52:53.677966 19049 heartbeater.cc:503] Master 127.27.54.126:40693 was elected leader, sending a full tablet report...
I20260807 06:52:53.680382 19259 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:52:53.683413 19280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:53.684850 19259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:52:53.685041 19259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "35a9b32904d84367a843b3c31550d1c8"
format_stamp: "Formatted at 2026-08-07 06:52:30 on dist-test-slave-3kk6"
server_key: "8e7b46e8ef5b7adc6e750a7c6f1c6d0c"
server_key_iv: "1fbd447a0bba2bd25f9d54bc9052a15d"
server_key_version: "encryptionkey@0"
I20260807 06:52:53.686127 19259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:52:53.709204 19259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:52:53.714219 19259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:52:53.714550 19259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:52:53.715778 19259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:52:53.717336 19259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:52:53.717435 19259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:53.717592 19259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:52:53.717691 19259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:52:53.755317 19259 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:38967
I20260807 06:52:53.755440 19436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:38967 every 8 connection(s)
I20260807 06:52:53.757323 19259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:52:53.760742 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 19259
Aug 07 06:52:53 dist-test-slave-3kk6 krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085573, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:52:53.779510 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.761902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.27.54.126:40693)
0807 06:52:53.762519 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.762525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.763209 (+ 684us) server_negotiation.cc:407] Connection header received
0807 06:52:53.763405 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.763410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.763482 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.763589 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:53.765970 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.767018 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.768669 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.769159 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.769297 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.772604 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:53.772609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:53.772613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:53.772645 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:53.776219 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.777138 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.777146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.777150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.777316 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.777796 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.777801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.777805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.778093 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:53.778328 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.778825 (+ 497us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.779096 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":318,"threads_started":1}
I20260807 06:52:53.782265 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.760091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:47407 (local address 127.27.54.126:40693)
0807 06:52:53.760413 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.760420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.761693 (+ 1273us) server_negotiation.cc:407] Connection header received
0807 06:52:53.764605 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.764610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.764689 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.764781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:53.766958 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.767885 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.769809 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.770367 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.770515 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.774106 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:53.774112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:53.774117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:53.774169 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:53.777654 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.778758 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.778764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.778767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.778937 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.779555 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.779559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.779562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.780026 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:53.780207 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.781602 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.781871 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260807 06:52:53.783471 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.760593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:47407)
0807 06:52:53.761503 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.761551 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.764254 (+ 2703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.765137 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.765152 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.765964 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:52:53.766754 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.766777 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.768045 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.768055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.769665 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.769679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.770620 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.770628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.770884 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.771555 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:52:53.771583 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:52:53.773874 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:52:53.777823 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:53.777838 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:53.777853 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:53.778531 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:53.779082 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:52:53.779089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:52:53.779095 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:52:53.779402 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:52:53.780186 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:52:53.780195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:52:53.781150 (+ 955us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.781470 (+ 320us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.782111 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":401,"threads_started":1}
I20260807 06:52:53.784875 19437 heartbeater.cc:347] Connected to a master server at 127.27.54.126:40693
I20260807 06:52:53.785238 19437 heartbeater.cc:464] Registering TS with master...
I20260807 06:52:53.786026 19437 heartbeater.cc:511] Master 127.27.54.126:40693 requested a full tablet report, sending...
I20260807 06:52:53.787722 18787 ts_manager.cc:194] Registered new tserver with Master: 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67:38967)
I20260807 06:52:53.788574 18787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:47407
I20260807 06:52:53.795080 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:52:53.802261 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.796183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.27.54.126:40693)
0807 06:52:53.796445 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.796453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.796767 (+ 314us) server_negotiation.cc:407] Connection header received
0807 06:52:53.796999 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.797004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.797168 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.797315 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:52:53.797714 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.798517 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.800206 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.800686 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.800798 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.801290 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:52:53.801594 (+ 304us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 06:52:53.801966 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34618: token expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 06:52:53.802433 19257 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34618: token expired
W20260807 06:52:53.803104 18671 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.54.126:40693)
I20260807 06:52:53.818164 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.803693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.27.54.126:40693)
0807 06:52:53.803857 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.803861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.803909 (+ 48us) server_negotiation.cc:407] Connection header received
0807 06:52:53.804108 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.804113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.804186 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.804297 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:53.805989 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.807020 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.808479 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.808907 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.809078 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.812371 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:53.812376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:53.812379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:53.812435 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:53.815394 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.816254 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.816258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.816260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.816392 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:53.816793 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:53.816796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:53.816798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:53.817125 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:53.817356 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.817702 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.817908 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 06:52:53.822005 18787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:52:53.829883 18787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:52:53.854251 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.843284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43423 (local address 127.0.0.1:36812)
0807 06:52:53.845398 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.845416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.845557 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.846411 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.846419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.846469 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:53.847090 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.847109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.849020 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.849028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.851449 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.851466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.853064 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.853070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.853283 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.853350 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.853491 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.853708 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1981,"mutex_wait_us":83,"spinlock_wait_cycles":4608,"thread_start_us":1167,"threads_started":1}
I20260807 06:52:53.855319 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.843969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.27.54.65:43423)
0807 06:52:53.844960 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.844969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.845684 (+ 715us) server_negotiation.cc:407] Connection header received
0807 06:52:53.845791 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.845797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.845971 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.846137 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:53.847375 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.848887 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.851641 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.852821 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.852977 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.853232 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.854753 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.854953 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":990,"threads_started":1}
I20260807 06:52:53.855672 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.843436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.27.54.66:39487)
0807 06:52:53.844459 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.844470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.845472 (+ 1002us) server_negotiation.cc:407] Connection header received
0807 06:52:53.845599 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.845609 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.845895 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.846116 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:53.847830 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.849462 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.852526 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.854123 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.854291 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.854504 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.855132 (+ 628us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.855322 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":611,"threads_started":1}
I20260807 06:52:53.855849 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.845416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:38967 (local address 127.0.0.1:39480)
0807 06:52:53.847068 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.847088 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.847206 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.847913 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.847920 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.847945 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:53.848301 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.848312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.850291 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.850298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.852617 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.852628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.854942 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.854952 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.855159 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.855196 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.855309 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.855463 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":1,"thread_start_us":1292,"threads_started":1}
I20260807 06:52:53.856348 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.846410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.27.54.67:38967)
0807 06:52:53.847378 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.847387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.847423 (+ 36us) server_negotiation.cc:407] Connection header received
0807 06:52:53.847514 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.847519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.847685 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.847916 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:53.848516 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.850117 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.852779 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.854364 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.854501 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.854751 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.855366 (+ 615us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.855886 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":802,"threads_started":1}
I20260807 06:52:53.856463 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.843240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:39487 (local address 127.0.0.1:50096)
0807 06:52:53.844984 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.845037 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.845243 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.846411 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.846419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.846468 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:53.847631 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.847645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.849664 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.849670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.852335 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.852352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.854714 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.854721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.854899 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.854936 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.855937 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.856082 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1486,"mutex_wait_us":888,"spinlock_wait_cycles":8832,"thread_start_us":1315,"threads_started":1}
I20260807 06:52:53.860054 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 9e87da5a70f64e20a6946a3c447baf8d (DEFAULT_TABLE table=test-table [id=276ca9382bb140538e99e53057633115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:53.860064 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 9e87da5a70f64e20a6946a3c447baf8d (DEFAULT_TABLE table=test-table [id=276ca9382bb140538e99e53057633115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:53.860065 18952 tablet_service.cc:1467] Processing CreateTablet for tablet 9e87da5a70f64e20a6946a3c447baf8d (DEFAULT_TABLE table=test-table [id=276ca9382bb140538e99e53057633115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:52:53.861385 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e87da5a70f64e20a6946a3c447baf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:53.861521 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e87da5a70f64e20a6946a3c447baf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:53.861552 18952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e87da5a70f64e20a6946a3c447baf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:52:53.875340 19455 tablet_bootstrap.cc:492] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Bootstrap starting.
I20260807 06:52:53.879530 19457 tablet_bootstrap.cc:492] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Bootstrap starting.
I20260807 06:52:53.880088 19455 tablet_bootstrap.cc:654] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:53.881695 19455 log.cc:824] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:53.882743 19456 tablet_bootstrap.cc:492] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Bootstrap starting.
I20260807 06:52:53.883834 19457 tablet_bootstrap.cc:654] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:53.885026 19457 log.cc:824] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:53.886770 19456 tablet_bootstrap.cc:654] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Neither blocks nor log segments found. Creating new log.
I20260807 06:52:53.886929 19455 tablet_bootstrap.cc:492] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: No bootstrap required, opened a new log
I20260807 06:52:53.887306 19455 ts_tablet_manager.cc:1397] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260807 06:52:53.888180 19456 log.cc:824] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Log is configured to *not* fsync() on all Append() calls
I20260807 06:52:53.888746 19457 tablet_bootstrap.cc:492] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: No bootstrap required, opened a new log
I20260807 06:52:53.889083 19457 ts_tablet_manager.cc:1397] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260807 06:52:53.892092 19456 tablet_bootstrap.cc:492] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: No bootstrap required, opened a new log
I20260807 06:52:53.892387 19456 ts_tablet_manager.cc:1397] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260807 06:52:53.895943 19455 raft_consensus.cc:348] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.896382 19455 raft_consensus.cc:374] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:53.896502 19455 raft_consensus.cc:729] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561a8cc9cc9f46d69dde810280185324, State: Initialized, Role: FOLLOWER
I20260807 06:52:53.897208 19455 consensus_queue.cc:260] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.901592 19457 raft_consensus.cc:348] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.902122 19457 raft_consensus.cc:374] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:53.902217 19457 raft_consensus.cc:729] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a9b32904d84367a843b3c31550d1c8, State: Initialized, Role: FOLLOWER
I20260807 06:52:53.902966 19457 consensus_queue.cc:260] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.903620 19456 raft_consensus.cc:348] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.904171 19456 raft_consensus.cc:374] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:52:53.904280 19456 raft_consensus.cc:729] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4072d3d4f2df42b7bfab3c24cf6eed85, State: Initialized, Role: FOLLOWER
I20260807 06:52:53.904439 19437 heartbeater.cc:503] Master 127.27.54.126:40693 was elected leader, sending a full tablet report...
I20260807 06:52:53.904937 19456 consensus_queue.cc:260] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.905174 19457 ts_tablet_manager.cc:1428] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260807 06:52:53.905694 19254 heartbeater.cc:503] Master 127.27.54.126:40693 was elected leader, sending a full tablet report...
I20260807 06:52:53.906281 19455 ts_tablet_manager.cc:1428] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260807 06:52:53.907109 19456 ts_tablet_manager.cc:1428] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260807 06:52:53.921545 19465 raft_consensus.cc:482] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:52:53.921898 19465 raft_consensus.cc:504] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.923771 19465 leader_election.cc:288] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423), 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67:38967)
I20260807 06:52:53.930621 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.924374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:33427 (local address 127.27.54.67:38967)
0807 06:52:53.924604 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.924613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.925386 (+ 773us) server_negotiation.cc:407] Connection header received
0807 06:52:53.925555 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.925559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.925621 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.925691 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:53.926184 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.926988 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.928796 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.929622 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.929834 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.929918 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.930200 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.930341 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:52:53.930938 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.924264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:43267 (local address 127.27.54.65:43423)
0807 06:52:53.924516 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.924524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.925305 (+ 781us) server_negotiation.cc:407] Connection header received
0807 06:52:53.925866 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.925869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.925924 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.926022 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:52:53.926591 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.927376 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.929127 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.929913 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.930102 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.930179 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.930481 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.930611 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 06:52:53.930984 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.924230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:43423 (local address 127.27.54.66:43267)
0807 06:52:53.925186 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.925204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.925396 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.926043 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.926052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.926077 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:53.926440 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.926449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.927505 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.927517 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.928961 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.928974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.930150 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.930157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.930316 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.930339 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.930443 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.930574 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":555,"threads_started":1}
I20260807 06:52:53.931502 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.924302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:38967 (local address 127.27.54.66:33427)
0807 06:52:53.925281 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0807 06:52:53.925296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:52:53.925420 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:52:53.925722 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:52:53.925726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:52:53.925744 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:52:53.926020 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.926028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.927128 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.927134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.928625 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:52:53.928638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.929863 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.929870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:52:53.930025 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.930050 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 06:52:53.930173 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 06:52:53.930328 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":608,"threads_started":1}
I20260807 06:52:53.932386 18981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e87da5a70f64e20a6946a3c447baf8d" candidate_uuid: "561a8cc9cc9f46d69dde810280185324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" is_pre_election: true
I20260807 06:52:53.932752 19370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e87da5a70f64e20a6946a3c447baf8d" candidate_uuid: "561a8cc9cc9f46d69dde810280185324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a9b32904d84367a843b3c31550d1c8" is_pre_election: true
I20260807 06:52:53.933094 18981 raft_consensus.cc:2454] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561a8cc9cc9f46d69dde810280185324 in term 0.
I20260807 06:52:53.933292 19370 raft_consensus.cc:2454] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561a8cc9cc9f46d69dde810280185324 in term 0.
I20260807 06:52:53.934016 19080 leader_election.cc:302] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4072d3d4f2df42b7bfab3c24cf6eed85, 561a8cc9cc9f46d69dde810280185324; no voters:
I20260807 06:52:53.934449 19465 raft_consensus.cc:2781] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:52:53.934619 19465 raft_consensus.cc:482] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:52:53.934777 19465 raft_consensus.cc:3037] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:53.937075 19465 raft_consensus.cc:504] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.937754 19465 leader_election.cc:288] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [CANDIDATE]: Term 1 election: Requested vote from peers 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423), 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67:38967)
I20260807 06:52:53.938223 18981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e87da5a70f64e20a6946a3c447baf8d" candidate_uuid: "561a8cc9cc9f46d69dde810280185324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
I20260807 06:52:53.938392 19370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e87da5a70f64e20a6946a3c447baf8d" candidate_uuid: "561a8cc9cc9f46d69dde810280185324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a9b32904d84367a843b3c31550d1c8"
I20260807 06:52:53.938472 18981 raft_consensus.cc:3037] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:53.938628 19370 raft_consensus.cc:3037] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:52:53.942058 18981 raft_consensus.cc:2454] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561a8cc9cc9f46d69dde810280185324 in term 1.
I20260807 06:52:53.942058 19370 raft_consensus.cc:2454] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561a8cc9cc9f46d69dde810280185324 in term 1.
I20260807 06:52:53.942617 19077 leader_election.cc:302] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35a9b32904d84367a843b3c31550d1c8, 4072d3d4f2df42b7bfab3c24cf6eed85, 561a8cc9cc9f46d69dde810280185324; no voters:
I20260807 06:52:53.943104 19465 raft_consensus.cc:2781] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:52:53.943607 19465 raft_consensus.cc:686] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 1 LEADER]: Becoming Leader. State: Replica: 561a8cc9cc9f46d69dde810280185324, State: Running, Role: LEADER
I20260807 06:52:53.944121 19465 consensus_queue.cc:237] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } }
I20260807 06:52:53.949261 18785 catalog_manager.cc:5495] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 561a8cc9cc9f46d69dde810280185324 (127.27.54.66). New cstate: current_term: 1 leader_uuid: "561a8cc9cc9f46d69dde810280185324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a8cc9cc9f46d69dde810280185324" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 } health_report { overall_health: UNKNOWN } } }
I20260807 06:52:53.990075 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:53.984358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.27.54.66:39487)
0807 06:52:53.984552 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:53.984557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:53.984576 (+ 19us) server_negotiation.cc:407] Connection header received
0807 06:52:53.984776 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:53.984780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:53.984888 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:53.985031 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:52:53.985389 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.986126 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.987764 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:53.988176 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:53.988276 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:53.988879 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:52:53.989480 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:52:53.989639 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:53.989708 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 06:52:53.989851 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 06:52:53.996212 19255 tablet.cc:2369] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:53.999778 18981 raft_consensus.cc:1261] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Refusing update from remote peer 561a8cc9cc9f46d69dde810280185324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:53.999765 19370 raft_consensus.cc:1261] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Refusing update from remote peer 561a8cc9cc9f46d69dde810280185324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:52:54.001022 19465 consensus_queue.cc:1035] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a9b32904d84367a843b3c31550d1c8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:52:54.001395 19472 consensus_queue.cc:1035] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4072d3d4f2df42b7bfab3c24cf6eed85" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 06:52:54.009096 19438 tablet.cc:2369] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:52:54.018445 19477 mvcc.cc:205] Tried to move back new op lower bound from 7315806511090233344 to 7315806510879174656. Current Snapshot: MvccSnapshot[applied={T|T < 7315806511090233344}]
I20260807 06:52:54.033552 18787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:54.033937 18787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:52:54.038012 18787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Sending DeleteTablet for 3 replicas of tablet 9e87da5a70f64e20a6946a3c447baf8d
I20260807 06:52:54.039536 18952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e87da5a70f64e20a6946a3c447baf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:54 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:36812
I20260807 06:52:54.039570 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e87da5a70f64e20a6946a3c447baf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:54 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:39480
I20260807 06:52:54.039573 19150 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e87da5a70f64e20a6946a3c447baf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:52:54 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:50096
I20260807 06:52:54.041095 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 18836
I20260807 06:52:54.055162 19492 tablet_replica.cc:330] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: stopping tablet replica
I20260807 06:52:54.055696 19492 raft_consensus.cc:2229] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:54.056236 19492 raft_consensus.cc:2258] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:54.059834 19492 ts_tablet_manager.cc:1905] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:54.060485 19490 tablet_replica.cc:330] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: stopping tablet replica
I20260807 06:52:54.060995 19495 tablet_replica.cc:330] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: stopping tablet replica
I20260807 06:52:54.061586 19495 raft_consensus.cc:2229] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:52:54.062129 19490 raft_consensus.cc:2229] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:52:54.062832 19490 raft_consensus.cc:2258] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:54.063737 19495 raft_consensus.cc:2258] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:52:54.066541 19490 ts_tablet_manager.cc:1905] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:54.067785 19495 ts_tablet_manager.cc:1905] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:52:54.072197 19492 ts_tablet_manager.cc:1918] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:54.072409 19492 log.cc:1196] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/9e87da5a70f64e20a6946a3c447baf8d
I20260807 06:52:54.073031 19492 ts_tablet_manager.cc:1939] T 9e87da5a70f64e20a6946a3c447baf8d P 4072d3d4f2df42b7bfab3c24cf6eed85: Deleting consensus metadata
I20260807 06:52:54.075075 19490 ts_tablet_manager.cc:1918] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:54.075387 19490 log.cc:1196] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/9e87da5a70f64e20a6946a3c447baf8d
I20260807 06:52:54.076061 19490 ts_tablet_manager.cc:1939] T 9e87da5a70f64e20a6946a3c447baf8d P 35a9b32904d84367a843b3c31550d1c8: Deleting consensus metadata
I20260807 06:52:54.078267 18765 catalog_manager.cc:4841] TS 35a9b32904d84367a843b3c31550d1c8 (127.27.54.67:38967): tablet 9e87da5a70f64e20a6946a3c447baf8d (table test-table [id=276ca9382bb140538e99e53057633115]) successfully deleted
I20260807 06:52:54.078753 18766 catalog_manager.cc:4841] TS 4072d3d4f2df42b7bfab3c24cf6eed85 (127.27.54.65:43423): tablet 9e87da5a70f64e20a6946a3c447baf8d (table test-table [id=276ca9382bb140538e99e53057633115]) successfully deleted
I20260807 06:52:54.082168 19495 ts_tablet_manager.cc:1918] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:52:54.082334 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:54.042705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.27.54.65:43423)
0807 06:52:54.042897 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:54.042903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:54.042927 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:52:54.043059 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:54.043064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:54.043150 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:54.043254 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:54.046075 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:54.047784 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:54.050686 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:54.051714 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:54.051905 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:54.068984 (+ 17079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:54.069006 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:54.069025 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:54.069085 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:54.073880 (+ 4795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:54.075176 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:54.075188 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:54.075191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:54.075379 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:54.079265 (+ 3886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:54.079273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:54.079278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:54.079844 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:54.080208 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:54.081631 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0807 06:52:54.081921 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:52:54.082370 19495 log.cc:1196] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/9e87da5a70f64e20a6946a3c447baf8d
I20260807 06:52:54.083091 19495 ts_tablet_manager.cc:1939] T 9e87da5a70f64e20a6946a3c447baf8d P 561a8cc9cc9f46d69dde810280185324: Deleting consensus metadata
I20260807 06:52:54.083199 19042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:54.084987 18765 catalog_manager.cc:4841] TS 561a8cc9cc9f46d69dde810280185324 (127.27.54.66:39487): tablet 9e87da5a70f64e20a6946a3c447baf8d (table test-table [id=276ca9382bb140538e99e53057633115]) successfully deleted
I20260807 06:52:54.184458 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 18836
I20260807 06:52:54.210232 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 19056
I20260807 06:52:54.228365 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:54.211171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.27.54.66:39487)
0807 06:52:54.211389 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:54.211393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:54.211536 (+ 143us) server_negotiation.cc:407] Connection header received
0807 06:52:54.211750 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:54.211754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:54.211832 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:54.211930 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:54.213876 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:54.214856 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:54.216788 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:54.217243 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:54.217352 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:54.221247 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:54.221266 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:54.221282 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:54.221349 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:54.225001 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:54.226027 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:54.226035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:54.226037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:54.226224 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:54.226743 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:54.226747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:54.226752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:54.227132 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:54.227426 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:54.227818 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 06:52:54.228010 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:52:54.229125 19244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:54.352113 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 19056
I20260807 06:52:54.379427 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 19259
I20260807 06:52:54.394771 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:52:54.380358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.27.54.67:38967)
0807 06:52:54.380587 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 06:52:54.380595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:52:54.380619 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:52:54.380847 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:52:54.380851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:52:54.380929 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:52:54.381042 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:52:54.382696 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:54.383454 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:54.384860 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:52:54.385273 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:52:54.385416 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:52:54.388334 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:52:54.388351 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:52:54.388368 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:52:54.388404 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:52:54.391586 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:54.392386 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:54.392393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:54.392396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:54.392563 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:52:54.392972 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:52:54.392977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:52:54.392980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:52:54.393395 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:52:54.393692 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 06:52:54.394091 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 06:52:54.394365 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:52:54.395524 19429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:54.491211 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 19259
I20260807 06:52:54.516701 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 18735
I20260807 06:52:54.517921 18827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:52:54.637400 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 18735
I20260807 06:53:14.671074 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40693
--webserver_interface=127.27.54.126
--webserver_port=44583
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:15.048673 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:15.049016 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:15.049191 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:15.056740 19500 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 06:53:15.056859 19500 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 06:53:15.056929 19500 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 06:53:15.057013 19500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:15.057111 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:15.057240 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:15.057331 19500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:15.057413 19500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:15.067817 19500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:15.069268 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:15.070767 19500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:15.083761 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:15.083739 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:15.083814 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:15.084712 19500 server_base.cc:1034] running on GCE node
I20260807 06:53:15.085659 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:15.087383 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:15.088752 19500 hybrid_clock.cc:648] HybridClock initialized: now 1786085595088589 us; error 60 us; skew 500 ppm
I20260807 06:53:15.089154 19500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:15 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085595, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:15.098313 19500 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:53:15.100237 19500 webserver.cc:460] Webserver started at http://127.27.54.126:44583/ using document root <none> and password file <none>
I20260807 06:53:15.101058 19500 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:15.101217 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:15.107262 19500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 06:53:15.110278 19516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:15.111910 19500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:15.112149 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "e0b8626855814334835047958d3033e2"
format_stamp: "Formatted at 2026-08-07 06:52:28 on dist-test-slave-3kk6"
server_key: "25bafb3be6057fd5304100b0a9f99631"
server_key_iv: "42e0cc069f885ac9e093fe1b50ce57ec"
server_key_version: "encryptionkey@0"
I20260807 06:53:15.113194 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:15.127194 19500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:15.132035 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:15.132392 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:15.152987 19500 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:40693
I20260807 06:53:15.153044 19585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:40693 every 8 connection(s)
I20260807 06:53:15.154773 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:15.159300 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 19500
I20260807 06:53:15.160467 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:43423
--local_ip_for_outbound_sockets=127.27.54.65
--tserver_master_addrs=127.27.54.126:40693
--webserver_port=45565
--webserver_interface=127.27.54.65
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:53:15.163590 19588 sys_catalog.cc:263] Verifying existing consensus state
I20260807 06:53:15.169011 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap starting.
I20260807 06:53:15.192148 19588 log.cc:824] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:15.208983 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 06:53:15.209645 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Bootstrap complete.
I20260807 06:53:15.219753 19588 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:53:15.220736 19588 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b8626855814334835047958d3033e2, State: Initialized, Role: FOLLOWER
I20260807 06:53:15.221481 19588 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:53:15.221738 19588 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:15.221879 19588 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:15.222070 19588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 2 FOLLOWER]: Advancing to term 3
I20260807 06:53:15.226191 19588 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:53:15.226819 19588 leader_election.cc:302] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b8626855814334835047958d3033e2; no voters:
I20260807 06:53:15.227607 19588 leader_election.cc:288] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 06:53:15.227768 19595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 06:53:15.229090 19595 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [term 3 LEADER]: Becoming Leader. State: Replica: e0b8626855814334835047958d3033e2, State: Running, Role: LEADER
I20260807 06:53:15.229732 19595 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } }
I20260807 06:53:15.230053 19588 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:15.232452 19597 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b8626855814334835047958d3033e2. Latest consensus state: current_term: 3 leader_uuid: "e0b8626855814334835047958d3033e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } } }
I20260807 06:53:15.232837 19597 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:15.232555 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0b8626855814334835047958d3033e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b8626855814334835047958d3033e2" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 40693 } } }
I20260807 06:53:15.232973 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:15.233800 19604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:15.259214 19604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:15.261462 19604 catalog_manager.cc:1259] Loaded cluster ID: 97fb94ab0de942319c7482163757aca7
I20260807 06:53:15.261729 19604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:15.267307 19604 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:15.269676 19604 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Loaded TSK: 0
I20260807 06:53:15.269989 19604 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0b8626855814334835047958d3033e2: Loaded TSK: 1
F20260807 06:53:15.271674 19604 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff7623489ad google::LogMessage::Fail() at ??:0
@ 0x7ff76234b88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff7623484de google::LogMessage::Flush() at ??:0
@ 0x7ff762349a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff7660ac7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff76609b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff777d771c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff777d76783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff777e022a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff777e020c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff777e4c5e9 std::function<>::operator()() at ??:0
@ 0x7ff777d7acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff777d78c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff777e00610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff777e00439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff764f94345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff764f9791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff764f97799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff764f72eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff76824c6db start_thread at ??:0
@ 0x7ff7611e261f clone at ??:0
W20260807 06:53:15.619040 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:15.619329 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:15.619429 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:15.626770 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:15.626894 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:15.627012 19591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:15.636871 19591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:43423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=45565
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:15.638113 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:15.639479 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:15.652806 19654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:15.652884 19657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:15.652884 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:15.653693 19591 server_base.cc:1034] running on GCE node
I20260807 06:53:15.654271 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:15.655272 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:15.656474 19591 hybrid_clock.cc:648] HybridClock initialized: now 1786085595656415 us; error 54 us; skew 500 ppm
I20260807 06:53:15.656814 19591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:15 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085595, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:15.660140 19591 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:53:15.661721 19591 webserver.cc:460] Webserver started at http://127.27.54.65:45565/ using document root <none> and password file <none>
I20260807 06:53:15.662412 19591 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:15.662525 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:15.668049 19591 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260807 06:53:15.670979 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:15.672370 19591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:53:15.672510 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "4072d3d4f2df42b7bfab3c24cf6eed85"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "d61d795500871f56d373ec8efbd9910b"
server_key_iv: "f81fe1b78abca24788d53960f779b05e"
server_key_version: "encryptionkey@0"
I20260807 06:53:15.673543 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:15.719487 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:15.724416 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:15.724701 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:15.725838 19591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:15.727335 19591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:15.727463 19591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:15.727602 19591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:15.727706 19591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:15.764006 19591 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:43423
I20260807 06:53:15.764104 19824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:43423 every 8 connection(s)
I20260807 06:53:15.765573 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:15.770514 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:15.768607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:41127)
0807 06:53:15.769383 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:15.769642 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":284,"threads_started":1}
W20260807 06:53:15.771281 19825 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40693 (0 consecutive failures): Network error: Failed to ping master at 127.27.54.126:40693: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
I20260807 06:53:15.772109 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:15.771799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:60989)
0807 06:53:15.771935 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:15.772004 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 06:53:15.772986 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:15.772643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:45039)
0807 06:53:15.772790 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:15.772877 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 06:53:15.773275 19825 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:53:15.776383 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 19591
I20260807 06:53:15.777381 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:39487
--local_ip_for_outbound_sockets=127.27.54.66
--tserver_master_addrs=127.27.54.126:40693
--webserver_port=46089
--webserver_interface=127.27.54.66
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:16.139791 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:16.140098 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:16.140264 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:16.147567 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:16.147696 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:16.147884 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:16.157716 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:39487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=46089
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:16.159073 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:16.160486 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:16.174350 19834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:16.174253 19840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:16.174288 19836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:16.175534 19828 server_base.cc:1034] running on GCE node
I20260807 06:53:16.176195 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:16.177163 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:16.178445 19828 hybrid_clock.cc:648] HybridClock initialized: now 1786085596178379 us; error 53 us; skew 500 ppm
I20260807 06:53:16.178838 19828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:16 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085596, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:16.182168 19828 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:53:16.183780 19828 webserver.cc:460] Webserver started at http://127.27.54.66:46089/ using document root <none> and password file <none>
I20260807 06:53:16.184497 19828 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:16.184653 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:16.190261 19828 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260807 06:53:16.193043 19849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:16.194377 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:53:16.194571 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "561a8cc9cc9f46d69dde810280185324"
format_stamp: "Formatted at 2026-08-07 06:52:29 on dist-test-slave-3kk6"
server_key: "789d4ce10597e42964d440c7fd2d2393"
server_key_iv: "0914709928d587212e9e6cae489da3dc"
server_key_version: "encryptionkey@0"
I20260807 06:53:16.195665 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:16.250200 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:16.254709 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:16.255033 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:16.256125 19828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:16.257539 19828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:16.257678 19828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:16.257825 19828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:16.257936 19828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:16.293769 19828 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:39487
I20260807 06:53:16.293871 20005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:39487 every 8 connection(s)
I20260807 06:53:16.295334 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:16.301054 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.298702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:43771)
0807 06:53:16.299753 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.300053 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":422,"threads_started":1}
W20260807 06:53:16.301746 20006 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40693 (0 consecutive failures): Network error: Failed to ping master at 127.27.54.126:40693: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
I20260807 06:53:16.302630 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.302307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:39189)
0807 06:53:16.302450 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.302519 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 06:53:16.303563 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.303233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:46745)
0807 06:53:16.303361 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.303452 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 06:53:16.303812 20006 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:53:16.303890 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 19828
I20260807 06:53:16.304605 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:38967
--local_ip_for_outbound_sockets=127.27.54.67
--tserver_master_addrs=127.27.54.126:40693
--webserver_port=35183
--webserver_interface=127.27.54.67
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:16.660653 20010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:16.660985 20010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:16.661087 20010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:16.668612 20010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:16.668749 20010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:16.668867 20010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:16.679003 20010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:38967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=35183
--tserver_master_addrs=127.27.54.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:16.680365 20010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:16.681710 20010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:16.695703 20017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:16.695878 20016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:16.695897 20021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:16.697292 20010 server_base.cc:1034] running on GCE node
I20260807 06:53:16.698081 20010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:16.699407 20010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:16.700663 20010 hybrid_clock.cc:648] HybridClock initialized: now 1786085596700602 us; error 68 us; skew 500 ppm
I20260807 06:53:16.701021 20010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:16 dist-test-slave-3kk6 krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085596, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:16.704758 20010 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:53:16.706298 20010 webserver.cc:460] Webserver started at http://127.27.54.67:35183/ using document root <none> and password file <none>
I20260807 06:53:16.707078 20010 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:16.707195 20010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:16.712848 20010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 06:53:16.716054 20031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:16.717585 20010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 06:53:16.717739 20010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "35a9b32904d84367a843b3c31550d1c8"
format_stamp: "Formatted at 2026-08-07 06:52:30 on dist-test-slave-3kk6"
server_key: "8e7b46e8ef5b7adc6e750a7c6f1c6d0c"
server_key_iv: "1fbd447a0bba2bd25f9d54bc9052a15d"
server_key_version: "encryptionkey@0"
I20260807 06:53:16.718849 20010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:16.740360 20010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:16.745149 20010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:16.745424 20010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:16.746487 20010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:16.748018 20010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:16.748111 20010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:16.748190 20010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:16.748248 20010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:16.775153 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.774081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:53475)
0807 06:53:16.774709 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.774826 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":255,"threads_started":1}
I20260807 06:53:16.786613 20010 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:38967
I20260807 06:53:16.786741 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:38967 every 8 connection(s)
I20260807 06:53:16.788364 20010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:16.789947 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 20010
I20260807 06:53:16.793759 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.791817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:45609)
0807 06:53:16.792596 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.792863 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":333,"threads_started":1}
W20260807 06:53:16.794553 20190 heartbeater.cc:644] Failed to heartbeat to 127.27.54.126:40693 (0 consecutive failures): Network error: Failed to ping master at 127.27.54.126:40693: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
I20260807 06:53:16.795511 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.795167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:44811)
0807 06:53:16.795314 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.795392 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 06:53:16.796454 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:16.796122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:55605)
0807 06:53:16.796255 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:16.796336 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:53:16.796837 20190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 06:53:17.305526 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:17.304461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:42313)
0807 06:53:17.305052 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:17.305160 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":282,"threads_started":1}
I20260807 06:53:17.777099 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:17.776187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:37763)
0807 06:53:17.776819 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:17.776914 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260807 06:53:17.798616 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:17.797659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:45479)
0807 06:53:17.798254 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:17.798366 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260807 06:53:18.309419 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:18.306905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:42291)
0807 06:53:18.308988 (+ 2083us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:18.309139 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1909,"thread_start_us":1762,"threads_started":1}
I20260807 06:53:18.778976 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:18.778116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:46579)
0807 06:53:18.778619 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:18.778782 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
I20260807 06:53:18.800635 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:18.799780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:53311)
0807 06:53:18.800365 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:18.800458 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
I20260807 06:53:19.311380 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:19.310539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:52757)
0807 06:53:19.311094 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:19.311184 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":326,"threads_started":1}
I20260807 06:53:19.780874 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:19.779996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:35885)
0807 06:53:19.780529 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:19.780648 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
I20260807 06:53:19.802562 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:19.801731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:58753)
0807 06:53:19.802241 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:19.802342 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260807 06:53:20.313401 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:20.312444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:55997)
0807 06:53:20.313132 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:20.313219 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":339,"threads_started":1}
I20260807 06:53:20.782842 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:20.781978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:50233)
0807 06:53:20.782506 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:20.782591 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":285,"threads_started":1}
I20260807 06:53:20.804431 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:20.803642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:54459)
0807 06:53:20.804143 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:20.804244 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":242,"threads_started":1}
I20260807 06:53:21.315407 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:21.314499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:57351)
0807 06:53:21.315093 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:21.315227 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":351,"threads_started":1}
I20260807 06:53:21.784860 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:21.783981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:54285)
0807 06:53:21.784544 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:21.784671 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":282,"threads_started":1}
I20260807 06:53:21.806514 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:21.805656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:45147)
0807 06:53:21.806216 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:21.806318 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":276,"threads_started":1}
I20260807 06:53:22.317291 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:22.316440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:34847)
0807 06:53:22.317002 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:22.317105 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":297,"threads_started":1}
I20260807 06:53:22.786912 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:22.785944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:33063)
0807 06:53:22.786448 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:22.786573 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":236,"threads_started":1}
I20260807 06:53:22.808398 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:22.807637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:40895)
0807 06:53:22.808126 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:22.808227 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":261,"threads_started":1}
I20260807 06:53:23.319352 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:23.318350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:35793)
0807 06:53:23.319050 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:23.319140 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":428,"threads_started":1}
I20260807 06:53:23.788910 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:23.787993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:56927)
0807 06:53:23.788573 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:23.788719 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":349,"threads_started":1}
I20260807 06:53:23.810349 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:23.809405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:43385)
0807 06:53:23.810000 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:23.810091 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":275,"threads_started":1}
I20260807 06:53:24.321223 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:24.320413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:52195)
0807 06:53:24.320917 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:24.321017 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260807 06:53:24.790560 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:24.789795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:49451)
0807 06:53:24.790302 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:24.790385 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":231,"threads_started":1}
I20260807 06:53:24.812162 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:24.811400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:58813)
0807 06:53:24.811902 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:24.811990 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260807 06:53:25.323118 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:25.322171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:44445)
0807 06:53:25.322831 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:25.322930 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":359,"threads_started":1}
I20260807 06:53:25.792423 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:25.791601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:34141)
0807 06:53:25.792156 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:25.792253 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":297,"threads_started":1}
I20260807 06:53:25.814153 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:25.813248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:38915)
0807 06:53:25.813829 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:25.813934 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":281,"threads_started":1}
I20260807 06:53:26.325016 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:26.324143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:49263)
0807 06:53:26.324735 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:26.324835 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":311,"threads_started":1}
I20260807 06:53:26.794301 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:26.793520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:50525)
0807 06:53:26.793995 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:26.794091 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":238,"threads_started":1}
I20260807 06:53:26.816134 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:26.815277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:49401)
0807 06:53:26.815869 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:26.815962 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":350,"threads_started":1}
I20260807 06:53:27.327158 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:27.326304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:47225)
0807 06:53:27.326870 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:27.326962 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
I20260807 06:53:27.796141 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:27.795322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:38663)
0807 06:53:27.795857 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:27.795948 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":266,"threads_started":1}
I20260807 06:53:27.817941 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:27.817118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:58645)
0807 06:53:27.817626 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:27.817732 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260807 06:53:28.328805 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:28.328099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:40573)
0807 06:53:28.328552 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:28.328638 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":237,"threads_started":1}
I20260807 06:53:28.798065 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:28.797239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:36899)
0807 06:53:28.797723 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:28.797830 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260807 06:53:28.819798 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:28.819073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:57981)
0807 06:53:28.819545 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:28.819634 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260807 06:53:29.330796 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:29.329874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:51109)
0807 06:53:29.330412 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:29.330523 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":275,"threads_started":1}
I20260807 06:53:29.800191 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:29.799193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:54715)
0807 06:53:29.799781 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:29.799891 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
I20260807 06:53:29.821746 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:29.820822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:48359)
0807 06:53:29.821417 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:29.821534 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":289,"threads_started":1}
I20260807 06:53:30.332564 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:30.331759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:44435)
0807 06:53:30.332262 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:30.332365 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260807 06:53:30.802584 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:30.801555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:46009)
0807 06:53:30.802151 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:30.802290 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":341,"threads_started":1}
I20260807 06:53:30.823779 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:30.822834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:46387)
0807 06:53:30.823456 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:30.823584 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":373,"threads_started":1}
I20260807 06:53:31.335050 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:31.334021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.66:40841)
0807 06:53:31.334728 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:31.334851 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":385,"threads_started":1}
I20260807 06:53:31.795149 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 19591
I20260807 06:53:31.804612 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:31.803574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.65:49625)
0807 06:53:31.804192 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:31.804372 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":352,"threads_started":1}
I20260807 06:53:31.816982 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:31.796450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.27.54.65:43423)
0807 06:53:31.797258 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:31.797281 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:31.797338 (+ 57us) server_negotiation.cc:407] Connection header received
0807 06:53:31.799676 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:31.799693 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:31.800193 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:31.800531 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:31.802095 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:31.803674 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:31.805462 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:31.806249 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:31.806503 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:31.809728 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:31.809737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:31.809757 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:31.809803 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:31.813129 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:31.814016 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:31.814025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:31.814033 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:31.814221 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:31.814704 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:31.814710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:31.814713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:31.815300 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:31.815631 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:31.816159 (+ 528us) server_negotiation.cc:299] Negotiation successful
0807 06:53:31.816659 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":402,"threads_started":1}
I20260807 06:53:31.818563 19820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:31.826161 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:31.825171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:40693 (local address 127.27.54.67:34537)
0807 06:53:31.825723 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:31.825896 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260807 06:53:31.907232 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 19591
I20260807 06:53:31.935424 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 19828
I20260807 06:53:31.955864 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:31.936551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.27.54.66:39487)
0807 06:53:31.937279 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:31.937293 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:31.937334 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:53:31.939159 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:31.939172 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:31.939649 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:31.940067 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:31.941735 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:31.943227 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:31.944619 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:31.945369 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:31.945595 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:31.948642 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:31.948651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:31.948668 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:31.948712 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:31.951886 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:31.952739 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:31.952748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:31.952754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:31.952970 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:31.953432 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:31.953439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:31.953442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:31.954059 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:31.954410 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:31.955020 (+ 610us) server_negotiation.cc:299] Negotiation successful
0807 06:53:31.955545 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":419,"threads_started":1}
I20260807 06:53:31.957481 19998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:32.045709 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 19828
I20260807 06:53:32.072885 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 20010
I20260807 06:53:32.093010 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:32.074028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.27.54.67:38967)
0807 06:53:32.074815 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:32.074829 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:32.074875 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:53:32.076550 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:32.076561 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:32.077012 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:32.077321 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:32.078875 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:32.080123 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:32.081441 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:32.082169 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:32.082329 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:32.085258 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:32.085266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:32.085285 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:32.085327 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:32.088836 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:32.089654 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:32.089664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:32.089671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:32.090020 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:32.090583 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:32.090589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:32.090594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:32.091429 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:32.091785 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:32.092239 (+ 454us) server_negotiation.cc:299] Negotiation successful
0807 06:53:32.092681 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":339,"threads_started":1}
I20260807 06:53:32.094592 20182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:32.183877 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 20010
2026-08-07T06:53:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65643 ms)
[----------] 30 tests from SecurityITest (324966 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:53:32 dist-test-slave-3kk6 krb5kdc[20275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:53:32 dist-test-slave-3kk6 krb5kdc[20275](info): set up 2 sockets
Aug 07 06:53:32 dist-test-slave-3kk6 krb5kdc[20275](info): commencing operation
krb5kdc: starting...
W20260807 06:53:33.536952 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:53:33 dist-test-slave-3kk6 krb5kdc[20275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:53:33.895346 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:36213
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:34.261790 20290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:34.262172 20290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:34.262338 20290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:34.269644 20290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:34.269812 20290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:34.269915 20290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:34.270026 20290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:34.270115 20290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:34.280467 20290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:36213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:34.281908 20290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:34.283300 20290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:34.296140 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:34.296123 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:34.296201 20301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:34.297149 20290 server_base.cc:1034] running on GCE node
I20260807 06:53:34.298049 20290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:34.299640 20290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:34.300863 20290 hybrid_clock.cc:648] HybridClock initialized: now 1786085614300799 us; error 54 us; skew 500 ppm
I20260807 06:53:34.301267 20290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:34 dist-test-slave-3kk6 krb5kdc[20275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085614, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:34.307938 20290 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:53:34.309562 20290 webserver.cc:460] Webserver started at http://127.27.54.126:36253/ using document root <none> and password file <none>
I20260807 06:53:34.310240 20290 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:34.310387 20290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:34.310786 20290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:34.313936 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "6b3cd019b8854fa38b0281013a049c2a"
format_stamp: "Formatted at 2026-08-07 06:53:34 on dist-test-slave-3kk6"
server_key: "6b286ba519ea9824ae3c72764c65ca24"
server_key_iv: "80526ad650140073673787a8d31f7bf0"
server_key_version: "encryptionkey@0"
I20260807 06:53:34.314771 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "6b3cd019b8854fa38b0281013a049c2a"
format_stamp: "Formatted at 2026-08-07 06:53:34 on dist-test-slave-3kk6"
server_key: "6b286ba519ea9824ae3c72764c65ca24"
server_key_iv: "80526ad650140073673787a8d31f7bf0"
server_key_version: "encryptionkey@0"
I20260807 06:53:34.320042 20290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 06:53:34.323626 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:34.325089 20290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:53:34.325251 20290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "6b3cd019b8854fa38b0281013a049c2a"
format_stamp: "Formatted at 2026-08-07 06:53:34 on dist-test-slave-3kk6"
server_key: "6b286ba519ea9824ae3c72764c65ca24"
server_key_iv: "80526ad650140073673787a8d31f7bf0"
server_key_version: "encryptionkey@0"
I20260807 06:53:34.325443 20290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:34.352221 20290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:34.356943 20290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:34.357213 20290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:34.377677 20290 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:36213
I20260807 06:53:34.377748 20380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:36213 every 8 connection(s)
I20260807 06:53:34.379225 20290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:34.382114 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 20290
I20260807 06:53:34.382370 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:34.383131 27865 external_mini_cluster.cc:1428] Setting key 4102418f33c0b20e8416585c664fe00e
I20260807 06:53:34.383296 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:34.398877 20383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a: Bootstrap starting.
Aug 07 06:53:34 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:34.403106 20383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:34.404538 20383 log.cc:824] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:34.408382 20383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a: No bootstrap required, opened a new log
I20260807 06:53:34.410051 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:34.385236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.27.54.126:36213)
0807 06:53:34.386084 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:34.386099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:34.386139 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:53:34.388144 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:34.388167 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:34.388563 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:34.388933 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:34.391080 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:34.392402 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:34.394356 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:34.395166 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:34.395314 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:34.400594 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:34.400609 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:34.400628 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:34.400709 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:34.404917 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:34.406001 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:34.406008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:34.406014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:34.406264 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:34.406737 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:34.406742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:34.406745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:34.407461 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:34.407771 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:34.408189 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 06:53:34.408653 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":386,"threads_started":1}
I20260807 06:53:34.417520 20383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3cd019b8854fa38b0281013a049c2a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36213 } }
I20260807 06:53:34.417971 20383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:34.418061 20383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3cd019b8854fa38b0281013a049c2a, State: Initialized, Role: FOLLOWER
I20260807 06:53:34.418573 20383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3cd019b8854fa38b0281013a049c2a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36213 } }
I20260807 06:53:34.418882 20383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:34.419026 20383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:34.419158 20383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:34.421816 20383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3cd019b8854fa38b0281013a049c2a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36213 } }
I20260807 06:53:34.422293 20383 leader_election.cc:302] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b3cd019b8854fa38b0281013a049c2a; no voters:
I20260807 06:53:34.422942 20383 leader_election.cc:288] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:34.423110 20393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:34.423839 20393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [term 1 LEADER]: Becoming Leader. State: Replica: 6b3cd019b8854fa38b0281013a049c2a, State: Running, Role: LEADER
I20260807 06:53:34.424437 20393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3cd019b8854fa38b0281013a049c2a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36213 } }
I20260807 06:53:34.425101 20383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:34.427654 20395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3cd019b8854fa38b0281013a049c2a. Latest consensus state: current_term: 1 leader_uuid: "6b3cd019b8854fa38b0281013a049c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3cd019b8854fa38b0281013a049c2a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36213 } } }
I20260807 06:53:34.427918 20394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b3cd019b8854fa38b0281013a049c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3cd019b8854fa38b0281013a049c2a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36213 } } }
I20260807 06:53:34.428156 20395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:34.428247 20394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:34.434271 20407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:34.439211 20407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:34.448712 20407 catalog_manager.cc:1347] Generated new cluster ID: 1028664322224e90886fba349f2c0306
I20260807 06:53:34.448863 20407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:34.475224 20407 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:34.477393 20407 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:34.490378 20407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a: Generated new TSK 0
I20260807 06:53:34.491297 20407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:53:34.578364 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36213
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:34.945683 20424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:34.946019 20424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:34.946118 20424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:34.954030 20424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:34.954180 20424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:34.954308 20424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:34.964917 20424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:34.966312 20424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:34.967729 20424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:34.982167 20430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:34.982051 20431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:34.982574 20434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:34.983493 20424 server_base.cc:1034] running on GCE node
I20260807 06:53:34.984174 20424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:34.985220 20424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:34.986470 20424 hybrid_clock.cc:648] HybridClock initialized: now 1786085614986420 us; error 46 us; skew 500 ppm
I20260807 06:53:34.986871 20424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:34 dist-test-slave-3kk6 krb5kdc[20275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085614, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:34.990475 20424 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:53:34.992144 20424 webserver.cc:460] Webserver started at http://127.27.54.65:45407/ using document root <none> and password file <none>
I20260807 06:53:34.992890 20424 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:34.993047 20424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:34.993399 20424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:34.996572 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "2f15a50983164990810d7a9e25036930"
format_stamp: "Formatted at 2026-08-07 06:53:34 on dist-test-slave-3kk6"
server_key: "dab0eae8ee2dd24e8c6b767780c76a73"
server_key_iv: "c7e5ca7b065709c1fb222173ac80b166"
server_key_version: "encryptionkey@0"
I20260807 06:53:34.997393 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "2f15a50983164990810d7a9e25036930"
format_stamp: "Formatted at 2026-08-07 06:53:34 on dist-test-slave-3kk6"
server_key: "dab0eae8ee2dd24e8c6b767780c76a73"
server_key_iv: "c7e5ca7b065709c1fb222173ac80b166"
server_key_version: "encryptionkey@0"
I20260807 06:53:35.003083 20424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260807 06:53:35.006909 20443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:35.008302 20424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:53:35.008499 20424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "2f15a50983164990810d7a9e25036930"
format_stamp: "Formatted at 2026-08-07 06:53:34 on dist-test-slave-3kk6"
server_key: "dab0eae8ee2dd24e8c6b767780c76a73"
server_key_iv: "c7e5ca7b065709c1fb222173ac80b166"
server_key_version: "encryptionkey@0"
I20260807 06:53:35.008731 20424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:35.040537 20424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:35.044865 20424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:35.045182 20424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:35.046197 20424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:35.047587 20424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:35.047698 20424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:35.047853 20424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:35.047938 20424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:35.083621 20424 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:34931
I20260807 06:53:35.083684 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:34931 every 8 connection(s)
I20260807 06:53:35.085050 20424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:35.095520 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 20424
I20260807 06:53:35.095779 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:53:35.096475 27865 external_mini_cluster.cc:1428] Setting key f09ac0c2c407f864a6415c5daaed4059
Aug 07 06:53:35 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085614, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:35.107013 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:35.087933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:44057 (local address 127.27.54.126:36213)
0807 06:53:35.088380 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:35.088385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:35.089427 (+ 1042us) server_negotiation.cc:407] Connection header received
0807 06:53:35.091447 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:35.091453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:35.091543 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:35.091703 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:35.093302 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.094039 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.096158 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.096653 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.096789 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:35.100709 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:35.100714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:35.100718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:35.100752 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:35.103661 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:35.104486 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:35.104491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:35.104493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:35.104622 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:35.105101 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:35.105105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:35.105107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:35.105381 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:35.105564 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:35.106316 (+ 752us) server_negotiation.cc:299] Negotiation successful
0807 06:53:35.106556 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":208,"threads_started":1}
I20260807 06:53:35.108028 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:35.088520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36213 (local address 127.27.54.65:44057)
0807 06:53:35.089231 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:35.089277 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:35.091144 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:35.091956 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:35.091964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:35.092525 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:35.093111 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:35.093137 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.094237 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.094241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:35.096007 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:35.096021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.096906 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.096914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:35.097160 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:35.097902 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:35.097933 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:35.100459 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:35.103834 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:35.103843 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:35.103862 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:35.104342 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:35.104752 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:35.104757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:35.104761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:35.104965 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:35.105528 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:35.105537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:35.105911 (+ 374us) client_negotiation.cc:769] Sending connection context
0807 06:53:35.106221 (+ 310us) client_negotiation.cc:240] Negotiation successful
0807 06:53:35.106820 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":297,"threads_started":1}
I20260807 06:53:35.109803 20597 heartbeater.cc:347] Connected to a master server at 127.27.54.126:36213
I20260807 06:53:35.110198 20597 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:35.111290 20597 heartbeater.cc:511] Master 127.27.54.126:36213 requested a full tablet report, sending...
I20260807 06:53:35.113616 20335 ts_manager.cc:194] Registered new tserver with Master: 2f15a50983164990810d7a9e25036930 (127.27.54.65:34931)
I20260807 06:53:35.115396 20335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:44057
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:53:35.174394 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36213
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:35.536351 20605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:35.536661 20605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:35.536761 20605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:35.544072 20605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:35.544205 20605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:35.544322 20605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:35.554394 20605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:35.555816 20605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:35.557137 20605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:35.571154 20619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:35.571247 20617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:35.571205 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:35.572646 20605 server_base.cc:1034] running on GCE node
I20260807 06:53:35.573344 20605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:35.574414 20605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:35.575654 20605 hybrid_clock.cc:648] HybridClock initialized: now 1786085615575604 us; error 48 us; skew 500 ppm
I20260807 06:53:35.576027 20605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:35 dist-test-slave-3kk6 krb5kdc[20275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:35.580333 20605 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:53:35.581995 20605 webserver.cc:460] Webserver started at http://127.27.54.66:44381/ using document root <none> and password file <none>
I20260807 06:53:35.582798 20605 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:35.582959 20605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:35.583324 20605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:35.586419 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "a0a41f54b1d24e029553f4033b6ab593"
format_stamp: "Formatted at 2026-08-07 06:53:35 on dist-test-slave-3kk6"
server_key: "a1cda551ad7ce7b3ee53ff0f2e6a0ae3"
server_key_iv: "7025eb8db5fcd4ad0462d73c14b2e80e"
server_key_version: "encryptionkey@0"
I20260807 06:53:35.587302 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "a0a41f54b1d24e029553f4033b6ab593"
format_stamp: "Formatted at 2026-08-07 06:53:35 on dist-test-slave-3kk6"
server_key: "a1cda551ad7ce7b3ee53ff0f2e6a0ae3"
server_key_iv: "7025eb8db5fcd4ad0462d73c14b2e80e"
server_key_version: "encryptionkey@0"
I20260807 06:53:35.592885 20605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 06:53:35.596773 20628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:35.598356 20605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:53:35.598575 20605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "a0a41f54b1d24e029553f4033b6ab593"
format_stamp: "Formatted at 2026-08-07 06:53:35 on dist-test-slave-3kk6"
server_key: "a1cda551ad7ce7b3ee53ff0f2e6a0ae3"
server_key_iv: "7025eb8db5fcd4ad0462d73c14b2e80e"
server_key_version: "encryptionkey@0"
I20260807 06:53:35.598899 20605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:35.613420 20605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:35.618201 20605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:35.618534 20605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:35.619680 20605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:35.621170 20605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:35.621286 20605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:35.621444 20605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:35.621524 20605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:35.658551 20605 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:41081
I20260807 06:53:35.658610 20777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:41081 every 8 connection(s)
I20260807 06:53:35.659991 20605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:35.661130 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 20605
I20260807 06:53:35.661363 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:53:35.661934 27865 external_mini_cluster.cc:1428] Setting key 8be78f7b8756cd99c479d525044020c9
Aug 07 06:53:35 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:35.686064 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:35.663084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:57451 (local address 127.27.54.126:36213)
0807 06:53:35.663595 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:35.663601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:35.664801 (+ 1200us) server_negotiation.cc:407] Connection header received
0807 06:53:35.667863 (+ 3062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:35.667867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:35.667942 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:35.668051 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:35.670362 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.671204 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.673855 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.674344 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.674462 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:35.679562 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:35.679568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:35.679571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:35.679614 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:35.682522 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:35.683442 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:35.683448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:35.683458 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:35.683599 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:35.684120 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:35.684124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:35.684126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:35.684427 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:35.684611 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:35.685407 (+ 796us) server_negotiation.cc:299] Negotiation successful
0807 06:53:35.685658 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260807 06:53:35.687223 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:35.663731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36213 (local address 127.27.54.66:57451)
0807 06:53:35.664597 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:35.664662 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:35.667503 (+ 2841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:35.668371 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:35.668385 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:35.669264 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:35.670113 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:35.670143 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.671380 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.671388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:35.673656 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:35.673674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:35.674616 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:35.674626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:35.675003 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:35.675904 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:35.675941 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:35.679213 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:35.682758 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:35.682769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:35.682791 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:35.683295 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:35.683745 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:35.683750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:35.683754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:35.683976 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:35.684586 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:35.684595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:35.684989 (+ 394us) client_negotiation.cc:769] Sending connection context
0807 06:53:35.685286 (+ 297us) client_negotiation.cc:240] Negotiation successful
0807 06:53:35.685884 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260807 06:53:35.689046 20780 heartbeater.cc:347] Connected to a master server at 127.27.54.126:36213
I20260807 06:53:35.689404 20780 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:35.690209 20780 heartbeater.cc:511] Master 127.27.54.126:36213 requested a full tablet report, sending...
I20260807 06:53:35.691901 20335 ts_manager.cc:194] Registered new tserver with Master: a0a41f54b1d24e029553f4033b6ab593 (127.27.54.66:41081)
I20260807 06:53:35.692842 20335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:57451
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:53:35.742476 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36213
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:53:36.118546 20597 heartbeater.cc:503] Master 127.27.54.126:36213 was elected leader, sending a full tablet report...
W20260807 06:53:36.136768 20790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:36.137087 20790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:36.137290 20790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:36.145130 20790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:36.145298 20790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:36.145486 20790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:36.155961 20790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:36213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:36.157480 20790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:36.158874 20790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:36.172653 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:36.172739 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:36.172750 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:36.173615 20790 server_base.cc:1034] running on GCE node
I20260807 06:53:36.174297 20790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:36.175386 20790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:36.176676 20790 hybrid_clock.cc:648] HybridClock initialized: now 1786085616176616 us; error 37 us; skew 500 ppm
I20260807 06:53:36.177042 20790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:36 dist-test-slave-3kk6 krb5kdc[20275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:36.180698 20790 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:53:36.182323 20790 webserver.cc:460] Webserver started at http://127.27.54.67:34661/ using document root <none> and password file <none>
I20260807 06:53:36.183168 20790 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:36.183339 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:36.183694 20790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:36.187214 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "a72f9c3b95c5440b890cc3eead6f0ed3"
format_stamp: "Formatted at 2026-08-07 06:53:36 on dist-test-slave-3kk6"
server_key: "174cf01726462833db670f2d578752e9"
server_key_iv: "b2147492962c1b27f9b2f68d989b59db"
server_key_version: "encryptionkey@0"
I20260807 06:53:36.188181 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "a72f9c3b95c5440b890cc3eead6f0ed3"
format_stamp: "Formatted at 2026-08-07 06:53:36 on dist-test-slave-3kk6"
server_key: "174cf01726462833db670f2d578752e9"
server_key_iv: "b2147492962c1b27f9b2f68d989b59db"
server_key_version: "encryptionkey@0"
I20260807 06:53:36.193990 20790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:53:36.197831 20813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:36.199344 20790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:36.199532 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "a72f9c3b95c5440b890cc3eead6f0ed3"
format_stamp: "Formatted at 2026-08-07 06:53:36 on dist-test-slave-3kk6"
server_key: "174cf01726462833db670f2d578752e9"
server_key_iv: "b2147492962c1b27f9b2f68d989b59db"
server_key_version: "encryptionkey@0"
I20260807 06:53:36.199764 20790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:36.229532 20790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:36.234517 20790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:36.234892 20790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:36.235980 20790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:36.237389 20790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:36.237465 20790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:36.237643 20790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:36.237697 20790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:36.272850 20790 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:45855
I20260807 06:53:36.272943 20965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:45855 every 8 connection(s)
I20260807 06:53:36.274241 20790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:36.279932 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 20790
I20260807 06:53:36.280140 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:53:36.280757 27865 external_mini_cluster.cc:1428] Setting key 3d66da3d0c6c0219f14d25077dad78c3
Aug 07 06:53:36 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:36.296530 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.276957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:48705 (local address 127.27.54.126:36213)
0807 06:53:36.277483 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.277488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.278330 (+ 842us) server_negotiation.cc:407] Connection header received
0807 06:53:36.280888 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.280895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.280982 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.281085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:36.283089 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.284059 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.285627 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.286044 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.286145 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.289792 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:36.289798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:36.289801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:36.289838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:36.292737 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.293576 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.293581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.293583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.293741 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.294221 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.294226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.294231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.294623 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:36.294824 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.295873 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.296117 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":271,"threads_started":1}
I20260807 06:53:36.297585 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.277437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36213 (local address 127.27.54.67:48705)
0807 06:53:36.278145 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:36.278185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:36.280523 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:36.281395 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:36.281404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:36.282088 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:36.282900 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.282923 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.284219 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.284223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.285485 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.285497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.286240 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.286244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.286451 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.287183 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:36.287206 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:36.289525 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:36.292914 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:36.292924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:36.292943 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:36.293427 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:36.293865 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:36.293871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:36.293875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:36.294086 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:36.294830 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:36.294839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:36.295305 (+ 466us) client_negotiation.cc:769] Sending connection context
0807 06:53:36.295696 (+ 391us) client_negotiation.cc:240] Negotiation successful
0807 06:53:36.296244 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":284,"threads_started":1}
I20260807 06:53:36.299120 20968 heartbeater.cc:347] Connected to a master server at 127.27.54.126:36213
I20260807 06:53:36.299455 20968 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:36.300236 20968 heartbeater.cc:511] Master 127.27.54.126:36213 requested a full tablet report, sending...
I20260807 06:53:36.301681 20335 ts_manager.cc:194] Registered new tserver with Master: a72f9c3b95c5440b890cc3eead6f0ed3 (127.27.54.67:45855)
I20260807 06:53:36.302558 20335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:48705
I20260807 06:53:36.309728 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:53:36.328606 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.314810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.27.54.126:36213)
0807 06:53:36.314988 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.314993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.315528 (+ 535us) server_negotiation.cc:407] Connection header received
0807 06:53:36.315778 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.315782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.315852 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.315965 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:36.317510 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.318299 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.319798 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.320219 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.320362 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.322994 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:36.323000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:36.323003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:36.323039 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:36.325575 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.326315 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.326319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.326322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.326483 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.326947 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.326952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.326955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.327304 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:36.327568 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.327970 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.328216 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:53:36.333086 20335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:53:36.340329 20335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:53:36.381778 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.355723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.27.54.66:41081)
0807 06:53:36.373291 (+ 17568us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.373301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.373326 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:53:36.373417 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.373423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.373633 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.373844 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:36.374697 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.375896 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.378695 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.380136 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.380273 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.380453 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.381273 (+ 820us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.381474 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17410,"thread_start_us":17305,"threads_started":1}
I20260807 06:53:36.381824 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.355689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34931 (local address 127.0.0.1:45274)
0807 06:53:36.371090 (+ 15401us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:36.371115 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:36.371281 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:36.372702 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:36.372714 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:36.372757 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:36.373266 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.373282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.375230 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.375236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.377665 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.377682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.380475 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.380481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.380957 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.380996 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 06:53:36.381127 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 06:53:36.381349 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15240,"mutex_wait_us":61,"thread_start_us":15032,"threads_started":1}
I20260807 06:53:36.382807 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.355365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:41081 (local address 127.0.0.1:56604)
0807 06:53:36.369337 (+ 13972us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:36.369367 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:36.369546 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:36.373998 (+ 4452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:36.374002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:36.374036 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:36.374487 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.374501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.376096 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.376101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.378481 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.378496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.380605 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.380611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.380920 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.380988 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 06:53:36.381119 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 06:53:36.382327 (+ 1208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13776,"mutex_wait_us":1006,"thread_start_us":14296,"threads_started":1}
I20260807 06:53:36.382866 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.356447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.27.54.65:34931)
0807 06:53:36.370470 (+ 14023us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.370484 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.371442 (+ 958us) server_negotiation.cc:407] Connection header received
0807 06:53:36.371603 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.371616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.372040 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.372311 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:36.373471 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.375058 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.377861 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.379906 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.380054 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.380326 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.382314 (+ 1988us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.382502 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13796,"thread_start_us":13615,"threads_started":1}
I20260807 06:53:36.383750 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.354137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.27.54.67:45855)
0807 06:53:36.373001 (+ 18864us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.373014 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.373055 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:53:36.373168 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.373178 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.373415 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.373635 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:36.374372 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.375847 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.380027 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.381594 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.381748 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.383151 (+ 1403us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.383230 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.383400 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18686,"thread_start_us":18525,"threads_started":1}
I20260807 06:53:36.384392 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.354050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:45855 (local address 127.0.0.1:60960)
0807 06:53:36.354915 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:36.354954 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:36.355139 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:36.373643 (+ 18504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:36.373648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:36.373679 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:36.374111 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.374126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.376013 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.376018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.379814 (+ 3796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.379829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.382172 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.382179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.382367 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.382401 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:53:36.382483 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 06:53:36.382618 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":102,"thread_start_us":415,"threads_started":1}
I20260807 06:53:36.387147 20500 tablet_service.cc:1467] Processing CreateTablet for tablet c0294c813f4f4a6e9225b9953581adec (DEFAULT_TABLE table=test-table [id=6250b4d224ae475da10ad741c35191a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:36.387167 20692 tablet_service.cc:1467] Processing CreateTablet for tablet c0294c813f4f4a6e9225b9953581adec (DEFAULT_TABLE table=test-table [id=6250b4d224ae475da10ad741c35191a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:36.388697 20500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0294c813f4f4a6e9225b9953581adec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:36.388695 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0294c813f4f4a6e9225b9953581adec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:36.390020 20874 tablet_service.cc:1467] Processing CreateTablet for tablet c0294c813f4f4a6e9225b9953581adec (DEFAULT_TABLE table=test-table [id=6250b4d224ae475da10ad741c35191a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:36.391542 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0294c813f4f4a6e9225b9953581adec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:36.403047 20995 tablet_bootstrap.cc:492] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Bootstrap starting.
I20260807 06:53:36.405519 20996 tablet_bootstrap.cc:492] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Bootstrap starting.
I20260807 06:53:36.407364 20995 tablet_bootstrap.cc:654] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:36.408799 20995 log.cc:824] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:36.409377 20996 tablet_bootstrap.cc:654] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:36.410873 20996 log.cc:824] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:36.411428 21000 tablet_bootstrap.cc:492] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Bootstrap starting.
I20260807 06:53:36.412926 20995 tablet_bootstrap.cc:492] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: No bootstrap required, opened a new log
I20260807 06:53:36.413276 20995 ts_tablet_manager.cc:1397] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 06:53:36.414891 21000 tablet_bootstrap.cc:654] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:36.416141 21000 log.cc:824] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:36.417781 20996 tablet_bootstrap.cc:492] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: No bootstrap required, opened a new log
I20260807 06:53:36.418085 20996 ts_tablet_manager.cc:1397] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260807 06:53:36.419111 21000 tablet_bootstrap.cc:492] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: No bootstrap required, opened a new log
I20260807 06:53:36.419375 21000 ts_tablet_manager.cc:1397] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260807 06:53:36.424198 20995 raft_consensus.cc:348] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.424687 20995 raft_consensus.cc:374] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:36.424788 20995 raft_consensus.cc:729] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f15a50983164990810d7a9e25036930, State: Initialized, Role: FOLLOWER
I20260807 06:53:36.425634 20995 consensus_queue.cc:260] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.427263 21000 raft_consensus.cc:348] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.427608 20995 ts_tablet_manager.cc:1428] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260807 06:53:36.427713 21000 raft_consensus.cc:374] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:36.427827 21000 raft_consensus.cc:729] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9c3b95c5440b890cc3eead6f0ed3, State: Initialized, Role: FOLLOWER
I20260807 06:53:36.428426 21000 consensus_queue.cc:260] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.428988 20996 raft_consensus.cc:348] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.429454 20996 raft_consensus.cc:374] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:36.429849 20968 heartbeater.cc:503] Master 127.27.54.126:36213 was elected leader, sending a full tablet report...
I20260807 06:53:36.430320 21000 ts_tablet_manager.cc:1428] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260807 06:53:36.429585 20996 raft_consensus.cc:729] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a41f54b1d24e029553f4033b6ab593, State: Initialized, Role: FOLLOWER
I20260807 06:53:36.431581 20996 consensus_queue.cc:260] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.432909 20780 heartbeater.cc:503] Master 127.27.54.126:36213 was elected leader, sending a full tablet report...
I20260807 06:53:36.433692 20996 ts_tablet_manager.cc:1428] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260807 06:53:36.433912 21004 raft_consensus.cc:482] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:36.434167 21004 raft_consensus.cc:504] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.436247 21004 leader_election.cc:288] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0a41f54b1d24e029553f4033b6ab593 (127.27.54.66:41081), 2f15a50983164990810d7a9e25036930 (127.27.54.65:34931)
I20260807 06:53:36.442850 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.436679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:41081 (local address 127.27.54.67:58373)
0807 06:53:36.436952 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:36.436978 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:36.437098 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:36.438033 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:36.438038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:36.438064 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:36.438258 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.438268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.439545 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.439551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.440886 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.440898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.442020 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.442027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.442223 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.442250 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:53:36.442361 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 06:53:36.442501 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260807 06:53:36.442847 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.437391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:58373 (local address 127.27.54.66:41081)
0807 06:53:36.437722 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.437730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.437750 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:53:36.437814 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.437819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.437920 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.438012 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:36.438403 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.439411 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.441052 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.441851 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.442020 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.442094 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.442387 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.442517 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260807 06:53:36.443499 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.437238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:39887 (local address 127.27.54.65:34931)
0807 06:53:36.437439 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.437445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.437890 (+ 445us) server_negotiation.cc:407] Connection header received
0807 06:53:36.438106 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.438112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.438203 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.438276 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:36.439178 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.439918 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.441632 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.442432 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.442629 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.442723 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.443098 (+ 375us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.443229 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:53:36.443939 20715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0294c813f4f4a6e9225b9953581adec" candidate_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a41f54b1d24e029553f4033b6ab593" is_pre_election: true
I20260807 06:53:36.444278 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.436705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:34931 (local address 127.27.54.67:39887)
0807 06:53:36.437760 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:36.437781 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:36.437925 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:36.438493 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:36.438500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:36.438531 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:36.438987 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.439002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.440080 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.440086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.441470 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:36.441481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.442678 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.442685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:36.442876 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.442900 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:53:36.443707 (+ 807us) client_negotiation.cc:240] Negotiation successful
0807 06:53:36.443870 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":21,"thread_start_us":660,"threads_started":1}
I20260807 06:53:36.444590 20715 raft_consensus.cc:2454] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9c3b95c5440b890cc3eead6f0ed3 in term 0.
I20260807 06:53:36.445401 20818 leader_election.cc:302] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a41f54b1d24e029553f4033b6ab593, a72f9c3b95c5440b890cc3eead6f0ed3; no voters:
I20260807 06:53:36.445286 20532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0294c813f4f4a6e9225b9953581adec" candidate_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f15a50983164990810d7a9e25036930" is_pre_election: true
I20260807 06:53:36.445859 21004 raft_consensus.cc:2781] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:53:36.445876 20532 raft_consensus.cc:2454] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9c3b95c5440b890cc3eead6f0ed3 in term 0.
I20260807 06:53:36.445988 21004 raft_consensus.cc:482] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:53:36.446111 21004 raft_consensus.cc:3037] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:36.448680 21004 raft_consensus.cc:504] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.449347 21004 leader_election.cc:288] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [CANDIDATE]: Term 1 election: Requested vote from peers a0a41f54b1d24e029553f4033b6ab593 (127.27.54.66:41081), 2f15a50983164990810d7a9e25036930 (127.27.54.65:34931)
I20260807 06:53:36.449839 20715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0294c813f4f4a6e9225b9953581adec" candidate_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a41f54b1d24e029553f4033b6ab593"
I20260807 06:53:36.449898 20532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0294c813f4f4a6e9225b9953581adec" candidate_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f15a50983164990810d7a9e25036930"
I20260807 06:53:36.450045 20715 raft_consensus.cc:3037] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:36.450155 20532 raft_consensus.cc:3037] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:36.452515 20715 raft_consensus.cc:2454] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9c3b95c5440b890cc3eead6f0ed3 in term 1.
I20260807 06:53:36.452548 20532 raft_consensus.cc:2454] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9c3b95c5440b890cc3eead6f0ed3 in term 1.
I20260807 06:53:36.452996 20818 leader_election.cc:302] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a41f54b1d24e029553f4033b6ab593, a72f9c3b95c5440b890cc3eead6f0ed3; no voters:
I20260807 06:53:36.453353 21004 raft_consensus.cc:2781] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:36.453864 21004 raft_consensus.cc:686] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 1 LEADER]: Becoming Leader. State: Replica: a72f9c3b95c5440b890cc3eead6f0ed3, State: Running, Role: LEADER
I20260807 06:53:36.454347 21004 consensus_queue.cc:237] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } }
I20260807 06:53:36.459630 20332 catalog_manager.cc:5495] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72f9c3b95c5440b890cc3eead6f0ed3 (127.27.54.67). New cstate: current_term: 1 leader_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72f9c3b95c5440b890cc3eead6f0ed3" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 45855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 } health_report { overall_health: UNKNOWN } } }
I20260807 06:53:36.493594 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.487874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.27.54.67:45855)
0807 06:53:36.488146 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.488151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.488171 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:53:36.488349 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.488353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.488432 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.488568 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:53:36.488966 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.489750 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.491286 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.491710 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.491812 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.492356 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:53:36.492895 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:53:36.493019 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.493177 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.493252 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 06:53:36.502349 20715 raft_consensus.cc:1261] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 1 FOLLOWER]: Refusing update from remote peer a72f9c3b95c5440b890cc3eead6f0ed3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:36.502401 20532 raft_consensus.cc:1261] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 1 FOLLOWER]: Refusing update from remote peer a72f9c3b95c5440b890cc3eead6f0ed3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:36.503497 21007 consensus_queue.cc:1035] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a41f54b1d24e029553f4033b6ab593" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:36.503974 21004 consensus_queue.cc:1035] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f15a50983164990810d7a9e25036930" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:36.518690 21013 mvcc.cc:205] Tried to move back new op lower bound from 7315806685183062016 to 7315806685001912320. Current Snapshot: MvccSnapshot[applied={T|T < 7315806685183062016}]
I20260807 06:53:36.535991 20332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:36.536366 20332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:36.540853 20332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b3cd019b8854fa38b0281013a049c2a: Sending DeleteTablet for 3 replicas of tablet c0294c813f4f4a6e9225b9953581adec
I20260807 06:53:36.542423 20500 tablet_service.cc:1514] Processing DeleteTablet for tablet c0294c813f4f4a6e9225b9953581adec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:36 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:45274
I20260807 06:53:36.542423 20692 tablet_service.cc:1514] Processing DeleteTablet for tablet c0294c813f4f4a6e9225b9953581adec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:36 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:56604
I20260807 06:53:36.542423 20874 tablet_service.cc:1514] Processing DeleteTablet for tablet c0294c813f4f4a6e9225b9953581adec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:36 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:60960
I20260807 06:53:36.545271 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 20424
I20260807 06:53:36.561509 21022 tablet_replica.cc:330] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: stopping tablet replica
I20260807 06:53:36.562096 21022 raft_consensus.cc:2229] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:36.562763 21022 raft_consensus.cc:2258] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:36.563700 21023 tablet_replica.cc:330] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: stopping tablet replica
I20260807 06:53:36.564186 21023 raft_consensus.cc:2229] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:36.564705 21023 raft_consensus.cc:2258] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:36.567255 21024 tablet_replica.cc:330] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: stopping tablet replica
I20260807 06:53:36.567782 21024 raft_consensus.cc:2229] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:53:36.568616 21024 raft_consensus.cc:2258] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:36.569015 21022 ts_tablet_manager.cc:1905] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:36.569442 21023 ts_tablet_manager.cc:1905] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 06:53:36 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:53:36.577059 21023 ts_tablet_manager.cc:1918] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:36.577265 21023 log.cc:1196] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/c0294c813f4f4a6e9225b9953581adec
I20260807 06:53:36.577230 21022 ts_tablet_manager.cc:1918] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:36.577404 21022 log.cc:1196] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/c0294c813f4f4a6e9225b9953581adec
I20260807 06:53:36.577883 21023 ts_tablet_manager.cc:1939] T c0294c813f4f4a6e9225b9953581adec P a0a41f54b1d24e029553f4033b6ab593: Deleting consensus metadata
I20260807 06:53:36.577975 21022 ts_tablet_manager.cc:1939] T c0294c813f4f4a6e9225b9953581adec P 2f15a50983164990810d7a9e25036930: Deleting consensus metadata
I20260807 06:53:36.580116 20313 catalog_manager.cc:4841] TS a0a41f54b1d24e029553f4033b6ab593 (127.27.54.66:41081): tablet c0294c813f4f4a6e9225b9953581adec (table test-table [id=6250b4d224ae475da10ad741c35191a9]) successfully deleted
I20260807 06:53:36.580236 20310 catalog_manager.cc:4841] TS 2f15a50983164990810d7a9e25036930 (127.27.54.65:34931): tablet c0294c813f4f4a6e9225b9953581adec (table test-table [id=6250b4d224ae475da10ad741c35191a9]) successfully deleted
I20260807 06:53:36.580726 21024 ts_tablet_manager.cc:1905] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:36.581836 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.547385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.27.54.65:34931)
0807 06:53:36.547649 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.547658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.551050 (+ 3392us) server_negotiation.cc:407] Connection header received
0807 06:53:36.552176 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.552183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.552289 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.552374 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:36.556221 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.557543 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.564316 (+ 6773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.565275 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.565482 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.572788 (+ 7306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:36.572807 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:36.572828 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:36.572884 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:36.577817 (+ 4933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.579073 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.579085 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.579089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.579298 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.579867 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.579872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.579876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.580537 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:36.580842 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.581208 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.581462 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 06:53:36.582479 20591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:36.591317 21024 ts_tablet_manager.cc:1918] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:36.591554 21024 log.cc:1196] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/c0294c813f4f4a6e9225b9953581adec
I20260807 06:53:36.592154 21024 ts_tablet_manager.cc:1939] T c0294c813f4f4a6e9225b9953581adec P a72f9c3b95c5440b890cc3eead6f0ed3: Deleting consensus metadata
I20260807 06:53:36.593868 20313 catalog_manager.cc:4841] TS a72f9c3b95c5440b890cc3eead6f0ed3 (127.27.54.67:45855): tablet c0294c813f4f4a6e9225b9953581adec (table test-table [id=6250b4d224ae475da10ad741c35191a9]) successfully deleted
I20260807 06:53:36.711865 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 20424
I20260807 06:53:36.748469 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 20605
Aug 07 06:53:36 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:53:36.769789 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.749511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.27.54.66:41081)
0807 06:53:36.749719 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.749724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.749746 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:53:36.749833 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.749837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.749917 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.750004 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:36.752083 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.752943 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.755317 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.755786 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.755905 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.761621 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:36.761644 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:36.761677 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:36.761736 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:36.766100 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.767065 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.767077 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.767081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.767261 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.767829 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.767835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.767839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.768388 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:36.768726 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.769087 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.769348 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:53:36.770706 20772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:36.865092 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 20605
I20260807 06:53:36.893558 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 20790
Aug 07 06:53:36 dist-test-slave-3kk6 krb5kdc[20275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:53:36.916009 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:36.894415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.27.54.67:45855)
0807 06:53:36.894607 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:36.894611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:36.894729 (+ 118us) server_negotiation.cc:407] Connection header received
0807 06:53:36.895022 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:36.895027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:36.895118 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:36.895284 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:36.897371 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.898221 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.900585 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:36.901049 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:36.901154 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:36.907685 (+ 6531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:36.907706 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:36.907729 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:36.907788 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:36.912493 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.913365 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.913380 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.913385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.913586 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:36.914081 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:36.914087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:36.914091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:36.914593 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:36.915051 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:36.915255 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 06:53:36.915527 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 06:53:36.916797 20961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:37.046514 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 20790
I20260807 06:53:37.073465 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 20290
I20260807 06:53:37.074833 20374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:37.187541 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 20290
2026-08-07T06:53:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5013 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T06:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:37Z Disabled control of system clock
I20260807 06:53:37.295125 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41311
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:41311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:53:37.649296 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:37.649647 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:37.649785 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:37.657475 21034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:37.657611 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:37.657691 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:37.657799 21034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:37.657886 21034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:37.667686 21034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:41311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:37.669166 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:37.671152 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:37.685083 21042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:37.685144 21045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:37.685169 21043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:37.685988 21034 server_base.cc:1034] running on GCE node
I20260807 06:53:37.686991 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:37.688609 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:37.689837 21034 hybrid_clock.cc:648] HybridClock initialized: now 1786085617689794 us; error 38 us; skew 500 ppm
I20260807 06:53:37.690248 21034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:37.692893 21034 webserver.cc:460] Webserver started at http://127.27.54.126:41191/ using document root <none> and password file <none>
I20260807 06:53:37.693650 21034 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:37.693778 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:37.694146 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:37.697566 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "a72dbe38255c4fb4a8bab1b50ed10181"
format_stamp: "Formatted at 2026-08-07 06:53:37 on dist-test-slave-3kk6"
server_key: "0e3f70d2bf60c47f7cc456725b31e1e9"
server_key_iv: "741cc300ff28b5ee676798df48fbab32"
server_key_version: "encryptionkey@0"
I20260807 06:53:37.698527 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "a72dbe38255c4fb4a8bab1b50ed10181"
format_stamp: "Formatted at 2026-08-07 06:53:37 on dist-test-slave-3kk6"
server_key: "0e3f70d2bf60c47f7cc456725b31e1e9"
server_key_iv: "741cc300ff28b5ee676798df48fbab32"
server_key_version: "encryptionkey@0"
I20260807 06:53:37.704370 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:53:37.708438 21054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:37.709882 21034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:53:37.710101 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "a72dbe38255c4fb4a8bab1b50ed10181"
format_stamp: "Formatted at 2026-08-07 06:53:37 on dist-test-slave-3kk6"
server_key: "0e3f70d2bf60c47f7cc456725b31e1e9"
server_key_iv: "741cc300ff28b5ee676798df48fbab32"
server_key_version: "encryptionkey@0"
I20260807 06:53:37.710364 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:37.733777 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:37.734794 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:37.735121 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:37.755277 21034 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:41311
I20260807 06:53:37.755314 21128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:41311 every 8 connection(s)
I20260807 06:53:37.756752 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:37.757586 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 21034
I20260807 06:53:37.757774 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:37.758379 27865 external_mini_cluster.cc:1428] Setting key 24155af8954aee5556ee7c58711bcbc3
I20260807 06:53:37.761566 21129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:37.775250 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:37.760581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.27.54.126:41311)
0807 06:53:37.761710 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:37.761726 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:37.761780 (+ 54us) server_negotiation.cc:407] Connection header received
0807 06:53:37.764194 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:37.764237 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:37.764872 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:37.765403 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:37.765957 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:37.768139 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:37.769705 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:37.770789 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:37.770981 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:37.772579 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:37.772597 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:37.772629 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:37.773361 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:37.773447 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:37.773469 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:37.773608 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:37.773862 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 06:53:37.774200 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":363,"threads_started":1}
I20260807 06:53:37.775975 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181: Bootstrap starting.
I20260807 06:53:37.779904 21129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:37.781252 21129 log.cc:824] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:37.784698 21129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181: No bootstrap required, opened a new log
I20260807 06:53:37.792637 21129 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41311 } }
I20260807 06:53:37.793072 21129 raft_consensus.cc:374] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:37.793195 21129 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72dbe38255c4fb4a8bab1b50ed10181, State: Initialized, Role: FOLLOWER
I20260807 06:53:37.793790 21129 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41311 } }
I20260807 06:53:37.794024 21129 raft_consensus.cc:388] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:37.794169 21129 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:37.794356 21129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:37.796782 21129 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41311 } }
I20260807 06:53:37.797362 21129 leader_election.cc:302] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a72dbe38255c4fb4a8bab1b50ed10181; no voters:
I20260807 06:53:37.798033 21129 leader_election.cc:288] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:37.798219 21135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:37.799517 21135 raft_consensus.cc:686] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [term 1 LEADER]: Becoming Leader. State: Replica: a72dbe38255c4fb4a8bab1b50ed10181, State: Running, Role: LEADER
I20260807 06:53:37.800252 21135 consensus_queue.cc:237] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41311 } }
I20260807 06:53:37.800449 21129 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:37.803682 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72dbe38255c4fb4a8bab1b50ed10181. Latest consensus state: current_term: 1 leader_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41311 } } }
I20260807 06:53:37.804174 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:37.803660 21136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72dbe38255c4fb4a8bab1b50ed10181" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41311 } } }
I20260807 06:53:37.804301 21136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:37.807893 21146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:37.813452 21146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:37.823500 21146 catalog_manager.cc:1347] Generated new cluster ID: 915d762947dc47999dd43e54942cbc66
I20260807 06:53:37.823652 21146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:37.835350 21146 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:37.836697 21146 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:37.845185 21146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181: Generated new TSK 0
I20260807 06:53:37.846091 21146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:53:37.856877 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41311
--builtin_ntp_servers=127.27.54.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:53:38.228703 21166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:38.228999 21166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:38.229156 21166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:38.236498 21166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:38.236611 21166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:38.236775 21166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:38.246666 21166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:38.248062 21166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:38.249938 21166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:38.263375 21178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:38.263317 21179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:38.263347 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:38.264472 21166 server_base.cc:1034] running on GCE node
I20260807 06:53:38.265100 21166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:38.266129 21166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:38.267359 21166 hybrid_clock.cc:648] HybridClock initialized: now 1786085618267326 us; error 60 us; skew 500 ppm
I20260807 06:53:38.267685 21166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:38.269970 21166 webserver.cc:460] Webserver started at http://127.27.54.65:33527/ using document root <none> and password file <none>
I20260807 06:53:38.270766 21166 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:38.270910 21166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:38.271237 21166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:38.274236 21166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "3dac2789cec84a25991d526fe54efdd8"
format_stamp: "Formatted at 2026-08-07 06:53:38 on dist-test-slave-3kk6"
server_key: "07153b85f3c50e5e1a6965379eab9d30"
server_key_iv: "4a657eee8b6bafb4c9bcf305afdfab67"
server_key_version: "encryptionkey@0"
I20260807 06:53:38.275132 21166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "3dac2789cec84a25991d526fe54efdd8"
format_stamp: "Formatted at 2026-08-07 06:53:38 on dist-test-slave-3kk6"
server_key: "07153b85f3c50e5e1a6965379eab9d30"
server_key_iv: "4a657eee8b6bafb4c9bcf305afdfab67"
server_key_version: "encryptionkey@0"
I20260807 06:53:38.280460 21166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:53:38.284130 21191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:38.285601 21166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:53:38.285753 21166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "3dac2789cec84a25991d526fe54efdd8"
format_stamp: "Formatted at 2026-08-07 06:53:38 on dist-test-slave-3kk6"
server_key: "07153b85f3c50e5e1a6965379eab9d30"
server_key_iv: "4a657eee8b6bafb4c9bcf305afdfab67"
server_key_version: "encryptionkey@0"
I20260807 06:53:38.286072 21166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:38.308287 21166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:38.309180 21166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:38.309475 21166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:38.310747 21166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:38.312399 21166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:38.312494 21166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:38.312579 21166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:38.312637 21166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:38.347838 21166 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:32939
I20260807 06:53:38.347939 21340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:32939 every 8 connection(s)
I20260807 06:53:38.349239 21166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:38.357060 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 21166
I20260807 06:53:38.357260 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:53:38.357759 27865 external_mini_cluster.cc:1428] Setting key 2d3f11afd9ef247430434f1db481b71a
I20260807 06:53:38.361234 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41311
--builtin_ntp_servers=127.27.54.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:53:38.363245 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:38.351929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:51661 (local address 127.27.54.126:41311)
0807 06:53:38.352400 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:38.352405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:38.353503 (+ 1098us) server_negotiation.cc:407] Connection header received
0807 06:53:38.355510 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:38.355517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:38.355617 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:38.355749 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:38.356891 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.357750 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.360063 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.360504 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.360615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:38.362437 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:38.362442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:38.362446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:38.362464 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:38.362506 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:38.362519 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:38.362592 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:38.362901 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 06:53:38.362984 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":255,"threads_started":1}
I20260807 06:53:38.364850 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:38.352411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41311 (local address 127.27.54.65:51661)
0807 06:53:38.353322 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:38.353368 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:38.355199 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:38.355999 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:38.356007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:38.356100 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:38.356702 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:38.356720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.358019 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.358025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:38.359913 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:38.359926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.360759 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.360767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:38.361116 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:38.362015 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:38.362022 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:38.362082 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:38.362215 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:38.362627 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:38.362641 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:38.362675 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:53:38.362887 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 06:53:38.363375 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
I20260807 06:53:38.366492 21341 heartbeater.cc:347] Connected to a master server at 127.27.54.126:41311
I20260807 06:53:38.366847 21341 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:38.367640 21341 heartbeater.cc:511] Master 127.27.54.126:41311 requested a full tablet report, sending...
I20260807 06:53:38.369807 21075 ts_manager.cc:194] Registered new tserver with Master: 3dac2789cec84a25991d526fe54efdd8 (127.27.54.65:32939)
I20260807 06:53:38.371409 21075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.65:51661
W20260807 06:53:38.740752 21347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:38.741068 21347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:38.741236 21347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:38.748656 21347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:38.748788 21347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:38.748975 21347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:38.758539 21347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:38.759949 21347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:38.761808 21347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:38.775660 21356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:38.775774 21360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:38.775684 21357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:38.776535 21347 server_base.cc:1034] running on GCE node
I20260807 06:53:38.777216 21347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:38.778214 21347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:38.779426 21347 hybrid_clock.cc:648] HybridClock initialized: now 1786085618779386 us; error 40 us; skew 500 ppm
I20260807 06:53:38.779758 21347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:38.782184 21347 webserver.cc:460] Webserver started at http://127.27.54.66:37699/ using document root <none> and password file <none>
I20260807 06:53:38.783032 21347 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:38.783128 21347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:38.783492 21347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:38.786484 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "5c8251ce351d4872a43cfe1257c26fde"
format_stamp: "Formatted at 2026-08-07 06:53:38 on dist-test-slave-3kk6"
server_key: "760a6a6ae691057a1fd94d77c8725ff0"
server_key_iv: "4f9aed4987cfc808d609e759df6042b0"
server_key_version: "encryptionkey@0"
I20260807 06:53:38.787447 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "5c8251ce351d4872a43cfe1257c26fde"
format_stamp: "Formatted at 2026-08-07 06:53:38 on dist-test-slave-3kk6"
server_key: "760a6a6ae691057a1fd94d77c8725ff0"
server_key_iv: "4f9aed4987cfc808d609e759df6042b0"
server_key_version: "encryptionkey@0"
I20260807 06:53:38.792752 21347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 06:53:38.796276 21369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:38.797641 21347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:53:38.797837 21347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "5c8251ce351d4872a43cfe1257c26fde"
format_stamp: "Formatted at 2026-08-07 06:53:38 on dist-test-slave-3kk6"
server_key: "760a6a6ae691057a1fd94d77c8725ff0"
server_key_iv: "4f9aed4987cfc808d609e759df6042b0"
server_key_version: "encryptionkey@0"
I20260807 06:53:38.798069 21347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:38.814947 21347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:38.815814 21347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:38.816143 21347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:38.817231 21347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:38.818866 21347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:38.819002 21347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:38.819130 21347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:38.819221 21347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:38.855376 21347 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:45177
I20260807 06:53:38.855441 21526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:45177 every 8 connection(s)
I20260807 06:53:38.856793 21347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:38.862854 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 21347
I20260807 06:53:38.863085 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:53:38.863654 27865 external_mini_cluster.cc:1428] Setting key 5c204040ccbb2f5035f3675de25875da
I20260807 06:53:38.867064 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41311
--builtin_ntp_servers=127.27.54.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:53:38.871889 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:38.859930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:45779 (local address 127.27.54.126:41311)
0807 06:53:38.860135 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:38.860140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:38.861631 (+ 1491us) server_negotiation.cc:407] Connection header received
0807 06:53:38.864209 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:38.864214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:38.864290 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:38.864380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:38.865568 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.866590 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.868400 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.869171 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.869339 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:38.870895 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:38.870905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:38.870912 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:38.870941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:38.870986 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:38.871005 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:38.871103 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:38.871360 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 06:53:38.871477 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:53:38.872854 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:38.860472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41311 (local address 127.27.54.66:45779)
0807 06:53:38.861443 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:38.861490 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:38.863831 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:38.864682 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:38.864691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:38.864792 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:38.865383 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:38.865401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.866815 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.866820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:38.868226 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:38.868239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:38.869440 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:38.869445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:38.869662 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:38.870483 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:38.870486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:38.870528 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:38.870618 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:38.871124 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:38.871134 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:38.871140 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 06:53:38.871318 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 06:53:38.871655 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":328,"threads_started":1}
I20260807 06:53:38.874548 21527 heartbeater.cc:347] Connected to a master server at 127.27.54.126:41311
I20260807 06:53:38.874924 21527 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:38.875962 21527 heartbeater.cc:511] Master 127.27.54.126:41311 requested a full tablet report, sending...
I20260807 06:53:38.877734 21075 ts_manager.cc:194] Registered new tserver with Master: 5c8251ce351d4872a43cfe1257c26fde (127.27.54.66:45177)
I20260807 06:53:38.878597 21075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.66:45779
W20260807 06:53:39.228020 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:39.228333 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:39.228422 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:39.235888 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:39.235999 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:39.236176 21533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:39.245647 21533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:41311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:39.247169 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:39.249027 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:39.263048 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:39.263053 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:39.263703 21543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:39.264314 21533 server_base.cc:1034] running on GCE node
I20260807 06:53:39.265045 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:39.266075 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:39.267323 21533 hybrid_clock.cc:648] HybridClock initialized: now 1786085619267271 us; error 45 us; skew 500 ppm
I20260807 06:53:39.267663 21533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:39.270155 21533 webserver.cc:460] Webserver started at http://127.27.54.67:46525/ using document root <none> and password file <none>
I20260807 06:53:39.270994 21533 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:39.271137 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:39.271468 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:39.274528 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "8440cb6b46ad49af9c70d21d50f64652"
format_stamp: "Formatted at 2026-08-07 06:53:39 on dist-test-slave-3kk6"
server_key: "cb57e034804aff790964c21e85c7a379"
server_key_iv: "373dadb50a06a16306ad6e297a07d191"
server_key_version: "encryptionkey@0"
I20260807 06:53:39.275471 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "8440cb6b46ad49af9c70d21d50f64652"
format_stamp: "Formatted at 2026-08-07 06:53:39 on dist-test-slave-3kk6"
server_key: "cb57e034804aff790964c21e85c7a379"
server_key_iv: "373dadb50a06a16306ad6e297a07d191"
server_key_version: "encryptionkey@0"
I20260807 06:53:39.280900 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:53:39.284796 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:39.286154 21533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:53:39.286348 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "8440cb6b46ad49af9c70d21d50f64652"
format_stamp: "Formatted at 2026-08-07 06:53:39 on dist-test-slave-3kk6"
server_key: "cb57e034804aff790964c21e85c7a379"
server_key_iv: "373dadb50a06a16306ad6e297a07d191"
server_key_version: "encryptionkey@0"
I20260807 06:53:39.286583 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:39.305904 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:39.306943 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:39.307297 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:39.308391 21533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:39.309963 21533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:39.310094 21533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:39.310240 21533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:39.310333 21533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:39.345688 21533 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:34269
I20260807 06:53:39.345758 21705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:34269 every 8 connection(s)
I20260807 06:53:39.347368 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:39.354132 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 21533
I20260807 06:53:39.354419 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:53:39.355073 27865 external_mini_cluster.cc:1428] Setting key e17dca1eaa60d553234ee834afed8953
I20260807 06:53:39.365598 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.350201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:52521 (local address 127.27.54.126:41311)
0807 06:53:39.350384 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.350388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.352198 (+ 1810us) server_negotiation.cc:407] Connection header received
0807 06:53:39.354247 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.354252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.354415 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.354553 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:39.355967 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.356979 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.358760 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.359166 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.359271 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.364857 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:39.364863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:39.364866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:39.364887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:39.364925 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:39.364938 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:39.365011 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.365244 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.365341 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 06:53:39.366741 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.350935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:41311 (local address 127.27.54.67:52521)
0807 06:53:39.352016 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:39.352053 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:39.353839 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:39.354865 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:39.354878 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:39.355005 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:39.355794 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.355815 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.357132 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.357136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.358570 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.358583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.359365 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.359371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.359579 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.364437 (+ 4858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:39.364441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:39.364489 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:39.364593 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:39.365031 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:39.365041 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:39.365048 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 06:53:39.365200 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 06:53:39.365548 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":418,"threads_started":1}
I20260807 06:53:39.368405 21706 heartbeater.cc:347] Connected to a master server at 127.27.54.126:41311
I20260807 06:53:39.368800 21706 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:39.369727 21706 heartbeater.cc:511] Master 127.27.54.126:41311 requested a full tablet report, sending...
I20260807 06:53:39.371274 21075 ts_manager.cc:194] Registered new tserver with Master: 8440cb6b46ad49af9c70d21d50f64652 (127.27.54.67:34269)
I20260807 06:53:39.372107 21075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.67:52521
I20260807 06:53:39.374496 21341 heartbeater.cc:503] Master 127.27.54.126:41311 was elected leader, sending a full tablet report...
I20260807 06:53:39.383858 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:53:39.394968 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.388153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.27.54.126:41311)
0807 06:53:39.388485 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.388493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.388970 (+ 477us) server_negotiation.cc:407] Connection header received
0807 06:53:39.389205 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.389215 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.389280 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.389395 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:39.389950 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.390860 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.392382 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.392807 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.392948 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.394165 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:39.394170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:39.394174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:39.394195 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:39.394226 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:39.394239 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:39.394335 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.394488 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.394581 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 06:53:39.399324 21075 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:53:39.406577 21075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:53:39.432973 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.421572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:32939 (local address 127.0.0.1:39142)
0807 06:53:39.423557 (+ 1985us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:39.423582 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:39.423734 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:39.425017 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:39.425025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:39.425057 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:39.425483 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.425494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.427771 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.427778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.430005 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.430017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.431944 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.431951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.432151 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.432201 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 06:53:39.432345 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 06:53:39.432552 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1846,"mutex_wait_us":36,"thread_start_us":1018,"threads_started":1}
I20260807 06:53:39.433595 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.419484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:45177 (local address 127.0.0.1:38954)
0807 06:53:39.422385 (+ 2901us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:39.422411 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:39.422571 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:39.425173 (+ 2602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:39.425185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:39.425240 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:39.425777 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.425814 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.428028 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.428033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.430372 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.430387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.432360 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.432367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.432581 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.432605 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:53:39.432723 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:53:39.432849 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"mutex_wait_us":2,"thread_start_us":3806,"threads_started":1}
I20260807 06:53:39.434214 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.422342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.27.54.65:32939)
0807 06:53:39.424407 (+ 2065us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.424414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.424436 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:53:39.424517 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.424522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.424760 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.425653 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:39.425853 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.427620 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.430238 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.431746 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.431895 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.432209 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.433650 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.433851 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1887,"thread_start_us":1746,"threads_started":1}
I20260807 06:53:39.434283 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.419795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.27.54.66:45177)
0807 06:53:39.423252 (+ 3457us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.423266 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.423305 (+ 39us) server_negotiation.cc:407] Connection header received
0807 06:53:39.423435 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.423447 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.423790 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.424826 (+ 1036us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:39.426034 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.427619 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.430598 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.432095 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.432248 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.432494 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.433083 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.433946 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3229,"thread_start_us":3091,"threads_started":1}
I20260807 06:53:39.434315 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.419289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:34269 (local address 127.0.0.1:53264)
0807 06:53:39.420241 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:39.420313 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:39.420532 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:39.427033 (+ 6501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:39.427037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:39.427062 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:39.427421 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.427431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.429416 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.429422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.431251 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.431263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.433515 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.433525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.433759 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.433789 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 06:53:39.433880 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 06:53:39.434030 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":672,"threads_started":1}
I20260807 06:53:39.434926 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.419548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.27.54.67:34269)
0807 06:53:39.426501 (+ 6953us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.426507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.426534 (+ 27us) server_negotiation.cc:407] Connection header received
0807 06:53:39.426617 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.426621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.426847 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.427039 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:39.427634 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.429262 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.431450 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.433299 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.433448 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.433735 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.434187 (+ 452us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.434579 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6797,"thread_start_us":6632,"threads_started":1}
I20260807 06:53:39.438218 21431 tablet_service.cc:1467] Processing CreateTablet for tablet ace0befd0f734e87b90b0a29f3f4352e (DEFAULT_TABLE table=test-table [id=3458767677bc4aef91eb41e7f23f04d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:39.438220 21254 tablet_service.cc:1467] Processing CreateTablet for tablet ace0befd0f734e87b90b0a29f3f4352e (DEFAULT_TABLE table=test-table [id=3458767677bc4aef91eb41e7f23f04d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:39.438220 21619 tablet_service.cc:1467] Processing CreateTablet for tablet ace0befd0f734e87b90b0a29f3f4352e (DEFAULT_TABLE table=test-table [id=3458767677bc4aef91eb41e7f23f04d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:39.439610 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace0befd0f734e87b90b0a29f3f4352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:39.439754 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace0befd0f734e87b90b0a29f3f4352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:39.439743 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace0befd0f734e87b90b0a29f3f4352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:39.453439 21732 tablet_bootstrap.cc:492] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Bootstrap starting.
I20260807 06:53:39.455528 21735 tablet_bootstrap.cc:492] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Bootstrap starting.
I20260807 06:53:39.456418 21736 tablet_bootstrap.cc:492] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Bootstrap starting.
I20260807 06:53:39.457774 21732 tablet_bootstrap.cc:654] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:39.459252 21732 log.cc:824] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:39.460058 21736 tablet_bootstrap.cc:654] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:39.460433 21735 tablet_bootstrap.cc:654] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:39.461320 21736 log.cc:824] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:39.462101 21735 log.cc:824] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:39.464649 21732 tablet_bootstrap.cc:492] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: No bootstrap required, opened a new log
I20260807 06:53:39.464763 21736 tablet_bootstrap.cc:492] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: No bootstrap required, opened a new log
I20260807 06:53:39.464988 21732 ts_tablet_manager.cc:1397] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260807 06:53:39.465065 21736 ts_tablet_manager.cc:1397] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 06:53:39.465317 21735 tablet_bootstrap.cc:492] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: No bootstrap required, opened a new log
I20260807 06:53:39.465587 21735 ts_tablet_manager.cc:1397] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260807 06:53:39.472667 21732 raft_consensus.cc:348] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.473165 21732 raft_consensus.cc:374] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:39.473317 21732 raft_consensus.cc:729] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8251ce351d4872a43cfe1257c26fde, State: Initialized, Role: FOLLOWER
I20260807 06:53:39.473956 21732 consensus_queue.cc:260] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.475164 21527 heartbeater.cc:503] Master 127.27.54.126:41311 was elected leader, sending a full tablet report...
I20260807 06:53:39.475936 21732 ts_tablet_manager.cc:1428] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260807 06:53:39.476125 21736 raft_consensus.cc:348] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.476220 21735 raft_consensus.cc:348] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.476667 21736 raft_consensus.cc:374] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:39.476696 21735 raft_consensus.cc:374] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:39.476801 21735 raft_consensus.cc:729] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8440cb6b46ad49af9c70d21d50f64652, State: Initialized, Role: FOLLOWER
I20260807 06:53:39.476801 21736 raft_consensus.cc:729] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dac2789cec84a25991d526fe54efdd8, State: Initialized, Role: FOLLOWER
I20260807 06:53:39.477422 21735 consensus_queue.cc:260] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.477422 21736 consensus_queue.cc:260] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.479391 21735 ts_tablet_manager.cc:1428] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260807 06:53:39.479722 21736 ts_tablet_manager.cc:1428] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260807 06:53:39.480224 21706 heartbeater.cc:503] Master 127.27.54.126:41311 was elected leader, sending a full tablet report...
I20260807 06:53:39.548291 21745 raft_consensus.cc:482] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:39.548692 21745 raft_consensus.cc:504] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.550639 21745 leader_election.cc:288] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c8251ce351d4872a43cfe1257c26fde (127.27.54.66:45177), 3dac2789cec84a25991d526fe54efdd8 (127.27.54.65:32939)
I20260807 06:53:39.556722 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.551301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:58195 (local address 127.27.54.65:32939)
0807 06:53:39.551469 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.551476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.551530 (+ 54us) server_negotiation.cc:407] Connection header received
0807 06:53:39.551821 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.551825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.551898 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.552016 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:39.552516 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.553244 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.554978 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.555780 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.555954 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.556026 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.556334 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.556459 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:53:39.556761 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.551236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:32939 (local address 127.27.54.67:58195)
0807 06:53:39.551425 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:39.551454 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:39.551647 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:39.552073 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:39.552078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:39.552110 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:39.552330 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.552341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.553395 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.553404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.554803 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.554816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.555951 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.555957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.556159 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.556183 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 06:53:39.556306 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 06:53:39.556452 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":4}
I20260807 06:53:39.557790 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace0befd0f734e87b90b0a29f3f4352e" candidate_uuid: "8440cb6b46ad49af9c70d21d50f64652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dac2789cec84a25991d526fe54efdd8" is_pre_election: true
I20260807 06:53:39.558355 21282 raft_consensus.cc:2454] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8440cb6b46ad49af9c70d21d50f64652 in term 0.
I20260807 06:53:39.558455 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.551250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:47045 (local address 127.27.54.66:45177)
0807 06:53:39.551469 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.551476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.552516 (+ 1040us) server_negotiation.cc:407] Connection header received
0807 06:53:39.552738 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.552743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.552819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.552935 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:39.553554 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.554313 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.556366 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.557460 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.557652 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.557727 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.558031 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.558172 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 06:53:39.558539 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.551258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:45177 (local address 127.27.54.67:47045)
0807 06:53:39.552387 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:39.552402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:39.552560 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:39.552954 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:39.552959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:39.552982 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:39.553358 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.553368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.554485 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.554491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.556147 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:39.556161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.557650 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.557656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:39.557854 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.557881 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 06:53:39.557999 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 06:53:39.558149 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":251,"thread_start_us":523,"threads_started":1}
I20260807 06:53:39.559154 21556 leader_election.cc:302] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dac2789cec84a25991d526fe54efdd8, 8440cb6b46ad49af9c70d21d50f64652; no voters:
I20260807 06:53:39.559573 21745 raft_consensus.cc:2781] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:53:39.559751 21745 raft_consensus.cc:482] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:53:39.559893 21745 raft_consensus.cc:3037] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:39.559854 21462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace0befd0f734e87b90b0a29f3f4352e" candidate_uuid: "8440cb6b46ad49af9c70d21d50f64652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8251ce351d4872a43cfe1257c26fde" is_pre_election: true
I20260807 06:53:39.560436 21462 raft_consensus.cc:2454] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8440cb6b46ad49af9c70d21d50f64652 in term 0.
I20260807 06:53:39.562865 21745 raft_consensus.cc:504] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.563614 21745 leader_election.cc:288] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [CANDIDATE]: Term 1 election: Requested vote from peers 5c8251ce351d4872a43cfe1257c26fde (127.27.54.66:45177), 3dac2789cec84a25991d526fe54efdd8 (127.27.54.65:32939)
I20260807 06:53:39.564054 21462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace0befd0f734e87b90b0a29f3f4352e" candidate_uuid: "8440cb6b46ad49af9c70d21d50f64652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8251ce351d4872a43cfe1257c26fde"
I20260807 06:53:39.564219 21282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace0befd0f734e87b90b0a29f3f4352e" candidate_uuid: "8440cb6b46ad49af9c70d21d50f64652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dac2789cec84a25991d526fe54efdd8"
I20260807 06:53:39.564308 21462 raft_consensus.cc:3037] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:39.564481 21282 raft_consensus.cc:3037] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:39.568174 21462 raft_consensus.cc:2454] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8440cb6b46ad49af9c70d21d50f64652 in term 1.
I20260807 06:53:39.568166 21282 raft_consensus.cc:2454] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8440cb6b46ad49af9c70d21d50f64652 in term 1.
I20260807 06:53:39.568687 21556 leader_election.cc:302] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dac2789cec84a25991d526fe54efdd8, 8440cb6b46ad49af9c70d21d50f64652; no voters:
I20260807 06:53:39.569022 21745 raft_consensus.cc:2781] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:39.569528 21745 raft_consensus.cc:686] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 1 LEADER]: Becoming Leader. State: Replica: 8440cb6b46ad49af9c70d21d50f64652, State: Running, Role: LEADER
I20260807 06:53:39.570048 21745 consensus_queue.cc:237] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } }
I20260807 06:53:39.575181 21075 catalog_manager.cc:5495] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8440cb6b46ad49af9c70d21d50f64652 (127.27.54.67). New cstate: current_term: 1 leader_uuid: "8440cb6b46ad49af9c70d21d50f64652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8440cb6b46ad49af9c70d21d50f64652" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 34269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 } health_report { overall_health: UNKNOWN } } }
W20260807 06:53:39.599243 21707 tablet.cc:2369] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:53:39.601833 21342 tablet.cc:2369] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:53:39.609026 21529 tablet.cc:2369] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:53:39.626202 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.620725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.27.54.67:34269)
0807 06:53:39.620961 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.620969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.621055 (+ 86us) server_negotiation.cc:407] Connection header received
0807 06:53:39.621246 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.621250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.621356 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.621489 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:53:39.621804 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.622563 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.624061 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.624493 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.624592 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.625060 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:53:39.625614 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:53:39.625781 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.625821 (+ 40us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.625888 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 06:53:39.634624 21282 raft_consensus.cc:1261] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 1 FOLLOWER]: Refusing update from remote peer 8440cb6b46ad49af9c70d21d50f64652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:39.634621 21462 raft_consensus.cc:1261] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 1 FOLLOWER]: Refusing update from remote peer 8440cb6b46ad49af9c70d21d50f64652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:39.635663 21747 consensus_queue.cc:1035] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8251ce351d4872a43cfe1257c26fde" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:39.636034 21745 consensus_queue.cc:1035] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dac2789cec84a25991d526fe54efdd8" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:39.645823 21753 mvcc.cc:205] Tried to move back new op lower bound from 7315806698013110272 to 7315806697763917824. Current Snapshot: MvccSnapshot[applied={T|T < 7315806698013110272}]
I20260807 06:53:39.648352 21755 mvcc.cc:205] Tried to move back new op lower bound from 7315806698013110272 to 7315806697763917824. Current Snapshot: MvccSnapshot[applied={T|T < 7315806698013110272}]
I20260807 06:53:39.664920 21073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:39.665262 21073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:39.669188 21073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a72dbe38255c4fb4a8bab1b50ed10181: Sending DeleteTablet for 3 replicas of tablet ace0befd0f734e87b90b0a29f3f4352e
I20260807 06:53:39.670545 21619 tablet_service.cc:1514] Processing DeleteTablet for tablet ace0befd0f734e87b90b0a29f3f4352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:39 UTC) from {username='slave'} at 127.0.0.1:53264
I20260807 06:53:39.670555 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet ace0befd0f734e87b90b0a29f3f4352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:39 UTC) from {username='slave'} at 127.0.0.1:38954
I20260807 06:53:39.670606 21254 tablet_service.cc:1514] Processing DeleteTablet for tablet ace0befd0f734e87b90b0a29f3f4352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:39 UTC) from {username='slave'} at 127.0.0.1:39142
I20260807 06:53:39.673249 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 21166
I20260807 06:53:39.683229 21766 tablet_replica.cc:330] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: stopping tablet replica
I20260807 06:53:39.683943 21768 tablet_replica.cc:330] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: stopping tablet replica
I20260807 06:53:39.683974 21767 tablet_replica.cc:330] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: stopping tablet replica
I20260807 06:53:39.684549 21768 raft_consensus.cc:2229] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:39.684564 21767 raft_consensus.cc:2229] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:39.685158 21768 raft_consensus.cc:2258] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:39.685178 21767 raft_consensus.cc:2258] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:39.686045 21766 raft_consensus.cc:2229] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:53:39.687259 21766 raft_consensus.cc:2258] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:39.687990 21768 ts_tablet_manager.cc:1905] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:39.688903 21767 ts_tablet_manager.cc:1905] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:39.692273 21766 ts_tablet_manager.cc:1905] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:39.696055 21768 ts_tablet_manager.cc:1918] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:39.696352 21768 log.cc:1196] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/ace0befd0f734e87b90b0a29f3f4352e
I20260807 06:53:39.697022 21768 ts_tablet_manager.cc:1939] T ace0befd0f734e87b90b0a29f3f4352e P 3dac2789cec84a25991d526fe54efdd8: Deleting consensus metadata
I20260807 06:53:39.698413 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.674965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.27.54.65:32939)
0807 06:53:39.675165 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.675177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.686388 (+ 11211us) server_negotiation.cc:407] Connection header received
0807 06:53:39.686749 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.686755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.686885 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.686971 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:39.687712 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.688904 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.691337 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.692080 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.692189 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.695209 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:39.695222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:39.695241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:39.696101 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:39.697394 (+ 1293us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:39.697435 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:39.697550 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.697872 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.698005 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:53:39.699592 21057 catalog_manager.cc:4841] TS 3dac2789cec84a25991d526fe54efdd8 (127.27.54.65:32939): tablet ace0befd0f734e87b90b0a29f3f4352e (table test-table [id=3458767677bc4aef91eb41e7f23f04d7]) successfully deleted
I20260807 06:53:39.699736 21767 ts_tablet_manager.cc:1918] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:39.699909 21767 log.cc:1196] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/ace0befd0f734e87b90b0a29f3f4352e
I20260807 06:53:39.699914 21335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:39.700510 21767 ts_tablet_manager.cc:1939] T ace0befd0f734e87b90b0a29f3f4352e P 5c8251ce351d4872a43cfe1257c26fde: Deleting consensus metadata
I20260807 06:53:39.702638 21056 catalog_manager.cc:4841] TS 5c8251ce351d4872a43cfe1257c26fde (127.27.54.66:45177): tablet ace0befd0f734e87b90b0a29f3f4352e (table test-table [id=3458767677bc4aef91eb41e7f23f04d7]) successfully deleted
I20260807 06:53:39.705770 21766 ts_tablet_manager.cc:1918] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:39.705994 21766 log.cc:1196] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/ace0befd0f734e87b90b0a29f3f4352e
I20260807 06:53:39.706518 21766 ts_tablet_manager.cc:1939] T ace0befd0f734e87b90b0a29f3f4352e P 8440cb6b46ad49af9c70d21d50f64652: Deleting consensus metadata
I20260807 06:53:39.709008 21055 catalog_manager.cc:4841] TS 8440cb6b46ad49af9c70d21d50f64652 (127.27.54.67:34269): tablet ace0befd0f734e87b90b0a29f3f4352e (table test-table [id=3458767677bc4aef91eb41e7f23f04d7]) successfully deleted
I20260807 06:53:39.813568 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 21166
I20260807 06:53:39.849647 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 21347
I20260807 06:53:39.858450 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.850623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.27.54.66:45177)
0807 06:53:39.850825 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.850829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.850852 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:53:39.851074 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.851078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.851191 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.851280 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:39.851818 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.852583 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.854516 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.855257 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.855387 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.856708 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:39.856719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:39.856734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:39.857567 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:39.857638 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:39.857663 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:39.857808 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.857956 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.858081 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:53:39.859241 21521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:39.951143 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 21347
I20260807 06:53:39.976862 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 21533
I20260807 06:53:39.984854 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:39.977767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.27.54.67:34269)
0807 06:53:39.977934 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:39.977938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:39.978144 (+ 206us) server_negotiation.cc:407] Connection header received
0807 06:53:39.978402 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:39.978409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:39.978496 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:39.978617 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:39.979106 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.979893 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.981341 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:39.981801 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:39.981977 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:39.983327 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:39.983338 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:39.983354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:39.983975 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:39.984029 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:39.984054 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:39.984184 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:39.984343 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 06:53:39.984455 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 06:53:39.985565 21699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:40.118592 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 21533
I20260807 06:53:40.145370 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 21034
I20260807 06:53:40.146569 21123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:40.257336 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 21034
2026-08-07T06:53:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3062 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:53:40 dist-test-slave-3kk6 krb5kdc[21779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:53:40 dist-test-slave-3kk6 krb5kdc[21779](info): set up 2 sockets
Aug 07 06:53:40 dist-test-slave-3kk6 krb5kdc[21779](info): commencing operation
krb5kdc: starting...
W20260807 06:53:41.605656 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:53:41 dist-test-slave-3kk6 krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:53:41.966069 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:37889
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:42.338461 21799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:42.338943 21799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:42.339135 21799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:42.346810 21799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:42.346958 21799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:42.347071 21799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:42.347182 21799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:42.347282 21799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:42.357785 21799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:37889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:42.359308 21799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:42.360698 21799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:42.373487 21807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:42.373448 21811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:42.373525 21809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:42.374511 21799 server_base.cc:1034] running on GCE node
I20260807 06:53:42.375326 21799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:42.376762 21799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:42.378014 21799 hybrid_clock.cc:648] HybridClock initialized: now 1786085622377970 us; error 59 us; skew 500 ppm
I20260807 06:53:42.378376 21799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:42 dist-test-slave-3kk6 krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085622, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:42.381817 21799 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:53:42.383581 21799 webserver.cc:460] Webserver started at http://127.27.54.126:35927/ using document root <none> and password file <none>
I20260807 06:53:42.384305 21799 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:42.384466 21799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:42.384814 21799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:42.388074 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "9a0d57625560463ab71788433e1477b7"
format_stamp: "Formatted at 2026-08-07 06:53:42 on dist-test-slave-3kk6"
server_key: "e5c39fbfe4f06a82933e81ee24b2ad48"
server_key_iv: "ed23508f01f4db19f59b0bfcc97f46f4"
server_key_version: "encryptionkey@0"
I20260807 06:53:42.388912 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "9a0d57625560463ab71788433e1477b7"
format_stamp: "Formatted at 2026-08-07 06:53:42 on dist-test-slave-3kk6"
server_key: "e5c39fbfe4f06a82933e81ee24b2ad48"
server_key_iv: "ed23508f01f4db19f59b0bfcc97f46f4"
server_key_version: "encryptionkey@0"
I20260807 06:53:42.394498 21799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:53:42.398207 21821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:42.399581 21799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:42.399822 21799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "9a0d57625560463ab71788433e1477b7"
format_stamp: "Formatted at 2026-08-07 06:53:42 on dist-test-slave-3kk6"
server_key: "e5c39fbfe4f06a82933e81ee24b2ad48"
server_key_iv: "ed23508f01f4db19f59b0bfcc97f46f4"
server_key_version: "encryptionkey@0"
I20260807 06:53:42.400049 21799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:42.413597 21799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:42.418027 21799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:42.418350 21799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:42.438481 21799 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:37889
I20260807 06:53:42.438501 21893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:37889 every 8 connection(s)
I20260807 06:53:42.440075 21799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:42.444025 21894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:42.444197 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 21799
I20260807 06:53:42.444420 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:42.444916 27865 external_mini_cluster.cc:1428] Setting key cfe9b595ceda40a8b914abc40e988762
I20260807 06:53:42.459915 21894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7: Bootstrap starting.
Aug 07 06:53:42 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:42.464097 21894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:42.465641 21894 log.cc:824] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:42.469695 21894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7: No bootstrap required, opened a new log
I20260807 06:53:42.471503 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:42.446944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.27.54.126:37889)
0807 06:53:42.447900 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:42.447916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:42.447970 (+ 54us) server_negotiation.cc:407] Connection header received
0807 06:53:42.449788 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:42.449826 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:42.450387 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:42.450918 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:42.452329 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:42.454325 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:42.455818 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:42.457005 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:42.457224 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:42.461795 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:42.461808 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:42.461829 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:42.461903 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:42.466196 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:42.467352 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:42.467363 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:42.467371 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:42.467598 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:42.468010 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:42.468015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:42.468018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:42.468792 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:42.469114 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:42.469525 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 06:53:42.470005 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":385,"threads_started":1}
I20260807 06:53:42.479137 21894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0d57625560463ab71788433e1477b7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37889 } }
I20260807 06:53:42.479547 21894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:42.479627 21894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0d57625560463ab71788433e1477b7, State: Initialized, Role: FOLLOWER
I20260807 06:53:42.480221 21894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0d57625560463ab71788433e1477b7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37889 } }
I20260807 06:53:42.480468 21894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:42.480611 21894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:42.480861 21894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:42.483444 21894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0d57625560463ab71788433e1477b7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37889 } }
I20260807 06:53:42.483937 21894 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0d57625560463ab71788433e1477b7; no voters:
I20260807 06:53:42.484490 21894 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:42.484701 21900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:42.485435 21900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0d57625560463ab71788433e1477b7, State: Running, Role: LEADER
I20260807 06:53:42.486140 21900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0d57625560463ab71788433e1477b7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37889 } }
I20260807 06:53:42.486928 21894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:42.489941 21902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0d57625560463ab71788433e1477b7. Latest consensus state: current_term: 1 leader_uuid: "9a0d57625560463ab71788433e1477b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0d57625560463ab71788433e1477b7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37889 } } }
I20260807 06:53:42.490446 21902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:42.490824 21901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0d57625560463ab71788433e1477b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0d57625560463ab71788433e1477b7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 37889 } } }
I20260807 06:53:42.491240 21901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:42.492815 21911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:42.497507 21911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:42.506618 21911 catalog_manager.cc:1347] Generated new cluster ID: 9a2d8dd8d1b34cfb9949ee0cd59c4018
I20260807 06:53:42.506774 21911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:42.521159 21911 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:42.523218 21911 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:42.536827 21911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7: Generated new TSK 0
I20260807 06:53:42.538051 21911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:53:42.627126 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:37889
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:43.002810 21925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:43.003150 21925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:43.003260 21925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:43.010914 21925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:43.011052 21925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:43.011178 21925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:43.021188 21925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:43.022538 21925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:43.023890 21925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:43.037767 21933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:43.037858 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:43.037843 21934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:43.038877 21925 server_base.cc:1034] running on GCE node
I20260807 06:53:43.039543 21925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:43.040642 21925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:43.041939 21925 hybrid_clock.cc:648] HybridClock initialized: now 1786085623041904 us; error 67 us; skew 500 ppm
I20260807 06:53:43.042307 21925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:43 dist-test-slave-3kk6 krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:43.045697 21925 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:53:43.047289 21925 webserver.cc:460] Webserver started at http://127.27.54.65:45405/ using document root <none> and password file <none>
I20260807 06:53:43.048031 21925 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:43.048188 21925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:43.048528 21925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:43.051728 21925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "84b770d7cab94576b1375a64fb926a4d"
format_stamp: "Formatted at 2026-08-07 06:53:43 on dist-test-slave-3kk6"
server_key: "6178f848b3af2f19baa5571cac473dcf"
server_key_iv: "f0e3b66f2f5204b59275e4bbc2a40861"
server_key_version: "encryptionkey@0"
I20260807 06:53:43.052577 21925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "84b770d7cab94576b1375a64fb926a4d"
format_stamp: "Formatted at 2026-08-07 06:53:43 on dist-test-slave-3kk6"
server_key: "6178f848b3af2f19baa5571cac473dcf"
server_key_iv: "f0e3b66f2f5204b59275e4bbc2a40861"
server_key_version: "encryptionkey@0"
I20260807 06:53:43.058343 21925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:53:43.062636 21949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:43.064317 21925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:43.064535 21925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "84b770d7cab94576b1375a64fb926a4d"
format_stamp: "Formatted at 2026-08-07 06:53:43 on dist-test-slave-3kk6"
server_key: "6178f848b3af2f19baa5571cac473dcf"
server_key_iv: "f0e3b66f2f5204b59275e4bbc2a40861"
server_key_version: "encryptionkey@0"
I20260807 06:53:43.064786 21925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:43.085835 21925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:43.091100 21925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:43.091444 21925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:43.092604 21925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:43.094722 21925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:43.094830 21925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 06:53:43.095012 21925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:43.095085 21925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:43.131400 21925 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:33991
I20260807 06:53:43.131450 22099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:33991 every 8 connection(s)
I20260807 06:53:43.132838 21925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:43.137104 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 21925
I20260807 06:53:43.137336 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:53:43.137782 27865 external_mini_cluster.cc:1428] Setting key 4b52d26299850533908f7d36866d17e5
Aug 07 06:53:43 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:43.154106 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:43.135532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:59943 (local address 127.27.54.126:37889)
0807 06:53:43.136061 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:43.136066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:43.137006 (+ 940us) server_negotiation.cc:407] Connection header received
0807 06:53:43.139199 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:43.139205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:43.139302 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:43.139426 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:43.140996 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.141705 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.143415 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.143855 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.144003 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:43.147761 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:43.147766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:43.147770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:43.147805 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:43.150781 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:43.151584 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:43.151590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:43.151593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:43.151746 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:43.152229 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:43.152238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:43.152240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:43.152524 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:43.152718 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:43.153467 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 06:53:43.153724 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":321,"threads_started":1}
I20260807 06:53:43.155283 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:43.136036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:37889 (local address 127.27.54.65:59943)
0807 06:53:43.136814 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:43.136856 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:43.138906 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:43.139684 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:43.139693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:43.140241 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:43.140818 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:43.140837 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.141883 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.141887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:43.143263 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:43.143276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.144109 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.144117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:43.144356 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:43.145103 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:43.145131 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:43.147498 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:43.150938 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:43.150948 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:43.150966 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:43.151440 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:43.151877 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:43.151882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:43.151885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:43.152094 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:43.152674 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:43.152684 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:43.153063 (+ 379us) client_negotiation.cc:769] Sending connection context
0807 06:53:43.153356 (+ 293us) client_negotiation.cc:240] Negotiation successful
0807 06:53:43.153938 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":278,"threads_started":1}
I20260807 06:53:43.157179 22100 heartbeater.cc:347] Connected to a master server at 127.27.54.126:37889
I20260807 06:53:43.157522 22100 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:43.158322 22100 heartbeater.cc:511] Master 127.27.54.126:37889 requested a full tablet report, sending...
I20260807 06:53:43.160923 21841 ts_manager.cc:194] Registered new tserver with Master: 84b770d7cab94576b1375a64fb926a4d (127.27.54.65:33991)
I20260807 06:53:43.163216 21841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:59943
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:53:43.217972 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:37889
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:43.583884 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:43.584203 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:43.584367 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:43.592216 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:43.592420 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:43.592623 22108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:43.603579 22108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:43.605131 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:43.606600 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:43.620425 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:43.620512 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:43.620707 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:43.621964 22108 server_base.cc:1034] running on GCE node
I20260807 06:53:43.622903 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:43.624024 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:43.625293 22108 hybrid_clock.cc:648] HybridClock initialized: now 1786085623625253 us; error 63 us; skew 500 ppm
I20260807 06:53:43.625665 22108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:43 dist-test-slave-3kk6 krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:43.629339 22108 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:53:43.631034 22108 webserver.cc:460] Webserver started at http://127.27.54.66:38923/ using document root <none> and password file <none>
I20260807 06:53:43.631774 22108 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:43.631896 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:43.632196 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:43.635639 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "773c6024ef844e439d5878398f8c47b2"
format_stamp: "Formatted at 2026-08-07 06:53:43 on dist-test-slave-3kk6"
server_key: "d1d7c3e6e3a3a787293b8544c257bce7"
server_key_iv: "76b9f4871e6dec212ad8ba71b777cfef"
server_key_version: "encryptionkey@0"
I20260807 06:53:43.636492 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "773c6024ef844e439d5878398f8c47b2"
format_stamp: "Formatted at 2026-08-07 06:53:43 on dist-test-slave-3kk6"
server_key: "d1d7c3e6e3a3a787293b8544c257bce7"
server_key_iv: "76b9f4871e6dec212ad8ba71b777cfef"
server_key_version: "encryptionkey@0"
I20260807 06:53:43.642335 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260807 06:53:43.646790 22133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:43.648350 22108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:53:43.648537 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "773c6024ef844e439d5878398f8c47b2"
format_stamp: "Formatted at 2026-08-07 06:53:43 on dist-test-slave-3kk6"
server_key: "d1d7c3e6e3a3a787293b8544c257bce7"
server_key_iv: "76b9f4871e6dec212ad8ba71b777cfef"
server_key_version: "encryptionkey@0"
I20260807 06:53:43.648738 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:43.671563 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:43.676787 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:43.677088 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:43.678246 22108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:43.679890 22108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:43.679975 22108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:43.680070 22108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:43.680147 22108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:43.717378 22108 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:41649
I20260807 06:53:43.717432 22284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:41649 every 8 connection(s)
I20260807 06:53:43.718971 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:43.725343 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 22108
I20260807 06:53:43.725584 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:53:43.726071 27865 external_mini_cluster.cc:1428] Setting key fbfde9ccc9898dad0311af6ee87d96cd
Aug 07 06:53:43 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085623, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:43.744508 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:43.721792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:50687 (local address 127.27.54.126:37889)
0807 06:53:43.722302 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:43.722307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:43.723500 (+ 1193us) server_negotiation.cc:407] Connection header received
0807 06:53:43.725603 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:43.725609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:43.725738 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:43.725890 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:43.728098 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.729158 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.730921 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.731735 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.731892 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:43.735984 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:43.735995 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:43.736002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:43.736054 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:43.740343 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:43.741391 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:43.741402 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:43.741408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:43.741624 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:43.742141 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:43.742158 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:43.742164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:43.742620 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:43.742848 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:43.743597 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 06:53:43.743925 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260807 06:53:43.745123 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:43.722308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:37889 (local address 127.27.54.66:50687)
0807 06:53:43.723275 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:43.723330 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:43.725250 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:43.726198 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:43.726212 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:43.727040 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:43.727896 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:43.727924 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.729371 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.729375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:43.730721 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:43.730736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:43.731969 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:43.731973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:43.732184 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:43.732971 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:43.733000 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:43.735656 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:43.740598 (+ 4942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:43.740607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:43.740630 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:43.741200 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:43.741761 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:43.741765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:43.741767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:43.741999 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:43.742815 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:43.742821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:43.743180 (+ 359us) client_negotiation.cc:769] Sending connection context
0807 06:53:43.743431 (+ 251us) client_negotiation.cc:240] Negotiation successful
0807 06:53:43.743880 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":474,"threads_started":1}
I20260807 06:53:43.746755 22285 heartbeater.cc:347] Connected to a master server at 127.27.54.126:37889
I20260807 06:53:43.747117 22285 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:43.748030 22285 heartbeater.cc:511] Master 127.27.54.126:37889 requested a full tablet report, sending...
I20260807 06:53:43.749881 21841 ts_manager.cc:194] Registered new tserver with Master: 773c6024ef844e439d5878398f8c47b2 (127.27.54.66:41649)
I20260807 06:53:43.750944 21841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:50687
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:53:43.808135 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:37889
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:53:44.166582 22100 heartbeater.cc:503] Master 127.27.54.126:37889 was elected leader, sending a full tablet report...
W20260807 06:53:44.181308 22295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:44.181631 22295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:44.181815 22295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:44.189448 22295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:44.189622 22295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:44.189827 22295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:44.200558 22295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:44.202032 22295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:44.203521 22295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:44.218082 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:44.218012 22305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:44.218071 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:44.219583 22295 server_base.cc:1034] running on GCE node
I20260807 06:53:44.220284 22295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:44.221323 22295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:44.222538 22295 hybrid_clock.cc:648] HybridClock initialized: now 1786085624222464 us; error 60 us; skew 500 ppm
I20260807 06:53:44.222914 22295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:44 dist-test-slave-3kk6 krb5kdc[21779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:44.226243 22295 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:53:44.227918 22295 webserver.cc:460] Webserver started at http://127.27.54.67:37973/ using document root <none> and password file <none>
I20260807 06:53:44.228657 22295 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:44.228816 22295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:44.229164 22295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:44.232885 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "7aab068b57714b6e833efc240406ffe8"
format_stamp: "Formatted at 2026-08-07 06:53:44 on dist-test-slave-3kk6"
server_key: "9bc56c41b1ef23146e74b797a6411e64"
server_key_iv: "29e9e6b306c20124d2cde0d4ceeda91b"
server_key_version: "encryptionkey@0"
I20260807 06:53:44.233788 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "7aab068b57714b6e833efc240406ffe8"
format_stamp: "Formatted at 2026-08-07 06:53:44 on dist-test-slave-3kk6"
server_key: "9bc56c41b1ef23146e74b797a6411e64"
server_key_iv: "29e9e6b306c20124d2cde0d4ceeda91b"
server_key_version: "encryptionkey@0"
I20260807 06:53:44.239490 22295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:53:44.243510 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:44.244987 22295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 06:53:44.245201 22295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "7aab068b57714b6e833efc240406ffe8"
format_stamp: "Formatted at 2026-08-07 06:53:44 on dist-test-slave-3kk6"
server_key: "9bc56c41b1ef23146e74b797a6411e64"
server_key_iv: "29e9e6b306c20124d2cde0d4ceeda91b"
server_key_version: "encryptionkey@0"
I20260807 06:53:44.245441 22295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:44.284495 22295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:44.289278 22295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:44.289633 22295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:44.290930 22295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:44.292559 22295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:44.292680 22295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:44.292843 22295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:44.292927 22295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:44.329463 22295 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:44489
I20260807 06:53:44.329558 22470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:44489 every 8 connection(s)
I20260807 06:53:44.330941 22295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:44.335489 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 22295
I20260807 06:53:44.335708 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:53:44.336222 27865 external_mini_cluster.cc:1428] Setting key b1ef466b9bc5093e445e9dbd8c6b344e
Aug 07 06:53:44 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085624, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:44.353294 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.333804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:35657 (local address 127.27.54.126:37889)
0807 06:53:44.334299 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.334304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.335388 (+ 1084us) server_negotiation.cc:407] Connection header received
0807 06:53:44.337725 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.337730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.337847 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.337981 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:44.339914 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.340721 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.342288 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.342765 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.342864 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.346521 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:44.346526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:44.346529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:44.346566 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:44.349495 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.350438 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.350444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.350447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.350693 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.351163 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.351168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.351172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.351564 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:44.351751 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.352688 (+ 937us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.352994 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260807 06:53:44.354357 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.334406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:37889 (local address 127.27.54.67:35657)
0807 06:53:44.335189 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.335251 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.337372 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.338311 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.338321 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.338982 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:44.339738 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.339762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.340869 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.340873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.342139 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.342152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.342967 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.342972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.343176 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.343874 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:44.343899 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:44.346285 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:44.349668 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:44.349676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:44.349691 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:44.350303 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:44.350835 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:44.350839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:44.350842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:44.351061 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:44.351759 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:44.351768 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:44.352211 (+ 443us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.352579 (+ 368us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.353089 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":360,"threads_started":1}
I20260807 06:53:44.355867 22471 heartbeater.cc:347] Connected to a master server at 127.27.54.126:37889
I20260807 06:53:44.356209 22471 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:44.357003 22471 heartbeater.cc:511] Master 127.27.54.126:37889 requested a full tablet report, sending...
I20260807 06:53:44.358580 21841 ts_manager.cc:194] Registered new tserver with Master: 7aab068b57714b6e833efc240406ffe8 (127.27.54.67:44489)
I20260807 06:53:44.359473 21841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:35657
I20260807 06:53:44.364811 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:53:44.391461 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.369238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.27.54.126:37889)
0807 06:53:44.369449 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.369456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.369905 (+ 449us) server_negotiation.cc:407] Connection header received
0807 06:53:44.370053 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.370057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.370120 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.370210 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:44.371762 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.372511 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.373931 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.374325 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.374422 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.384861 (+ 10439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:44.384871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:44.384877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:44.384943 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:44.388004 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.389036 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.389040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.389042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.389176 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.389768 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.389773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.389775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.390211 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:44.390493 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.391005 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.391173 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 06:53:44.395876 21841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:53:44.403362 21841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:53:44.428555 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.416608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47706)
0807 06:53:44.417700 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.417739 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.417918 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.419643 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.419652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.419689 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.420682 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.420697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.422833 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.422838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.425388 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.425404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.427198 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.427205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.427430 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.427498 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.427652 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.427921 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":171,"thread_start_us":699,"threads_started":1}
I20260807 06:53:44.428813 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.416964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.27.54.65:33991)
0807 06:53:44.418600 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.418612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.418645 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:53:44.418786 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.418794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.419085 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.419299 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.420973 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.422684 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.425575 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.426958 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.427101 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.427353 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.428270 (+ 917us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.428476 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1255,"threads_started":1}
I20260807 06:53:44.429189 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.419570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.27.54.66:41649)
0807 06:53:44.420640 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.420650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.420679 (+ 29us) server_negotiation.cc:407] Connection header received
0807 06:53:44.420776 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.420784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.421006 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.421815 (+ 809us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.422038 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.423279 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.426184 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.427689 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.427875 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.428076 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.428655 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.428836 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":784,"threads_started":1}
I20260807 06:53:44.429711 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.418162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:44489 (local address 127.0.0.1:54048)
0807 06:53:44.418906 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.418925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.419041 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.420315 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.420319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.420348 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.420971 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.420984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.423121 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.423128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.426030 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.426042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.427608 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.427615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.427836 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.427872 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.427968 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.428107 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":402,"thread_start_us":428,"threads_started":1}
I20260807 06:53:44.430197 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.418637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.27.54.67:44489)
0807 06:53:44.419691 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.419700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.419733 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:53:44.419837 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.419844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.420090 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.421171 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.421492 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.422974 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.426205 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.427374 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.427515 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.427756 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.429654 (+ 1898us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.429838 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":638,"threads_started":1}
I20260807 06:53:44.430193 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.419027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:41649 (local address 127.0.0.1:38182)
0807 06:53:44.419918 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.419941 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.420093 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.421219 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.421224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.421249 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.421677 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.421690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.423437 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.423444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.425987 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.426004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.427983 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.427989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.428222 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.428267 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.428369 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.428502 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":119,"thread_start_us":732,"threads_started":1}
I20260807 06:53:44.433384 22385 tablet_service.cc:1467] Processing CreateTablet for tablet e60f555a4f644ab0af2694a451373712 (DEFAULT_TABLE table=test-table [id=7697aceab9044792b8a9f28ad0f155b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:44.433750 22200 tablet_service.cc:1467] Processing CreateTablet for tablet e60f555a4f644ab0af2694a451373712 (DEFAULT_TABLE table=test-table [id=7697aceab9044792b8a9f28ad0f155b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:44.433494 22009 tablet_service.cc:1467] Processing CreateTablet for tablet e60f555a4f644ab0af2694a451373712 (DEFAULT_TABLE table=test-table [id=7697aceab9044792b8a9f28ad0f155b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:44.435036 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60f555a4f644ab0af2694a451373712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:44.435218 22200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60f555a4f644ab0af2694a451373712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:44.435899 22009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60f555a4f644ab0af2694a451373712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:44.456298 22505 tablet_bootstrap.cc:492] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Bootstrap starting.
I20260807 06:53:44.463514 22503 tablet_bootstrap.cc:492] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Bootstrap starting.
I20260807 06:53:44.463941 22505 tablet_bootstrap.cc:654] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:44.464846 22506 tablet_bootstrap.cc:492] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Bootstrap starting.
I20260807 06:53:44.466720 22505 log.cc:824] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:44.467666 22503 tablet_bootstrap.cc:654] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:44.469254 22503 log.cc:824] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:44.470068 22506 tablet_bootstrap.cc:654] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:44.471429 22506 log.cc:824] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:44.471621 22505 tablet_bootstrap.cc:492] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: No bootstrap required, opened a new log
I20260807 06:53:44.472105 22505 ts_tablet_manager.cc:1397] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260807 06:53:44.475634 22506 tablet_bootstrap.cc:492] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: No bootstrap required, opened a new log
I20260807 06:53:44.475943 22506 ts_tablet_manager.cc:1397] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260807 06:53:44.476692 22503 tablet_bootstrap.cc:492] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: No bootstrap required, opened a new log
I20260807 06:53:44.476996 22503 ts_tablet_manager.cc:1397] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260807 06:53:44.483907 22506 raft_consensus.cc:348] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.483775 22505 raft_consensus.cc:348] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.484313 22506 raft_consensus.cc:374] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:44.484314 22505 raft_consensus.cc:374] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:44.484423 22505 raft_consensus.cc:729] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aab068b57714b6e833efc240406ffe8, State: Initialized, Role: FOLLOWER
I20260807 06:53:44.484444 22506 raft_consensus.cc:729] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b770d7cab94576b1375a64fb926a4d, State: Initialized, Role: FOLLOWER
I20260807 06:53:44.485132 22506 consensus_queue.cc:260] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.485143 22505 consensus_queue.cc:260] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.486930 22471 heartbeater.cc:503] Master 127.27.54.126:37889 was elected leader, sending a full tablet report...
I20260807 06:53:44.487306 22506 ts_tablet_manager.cc:1428] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260807 06:53:44.487813 22505 ts_tablet_manager.cc:1428] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260807 06:53:44.489705 22503 raft_consensus.cc:348] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.490135 22503 raft_consensus.cc:374] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:44.490263 22503 raft_consensus.cc:729] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773c6024ef844e439d5878398f8c47b2, State: Initialized, Role: FOLLOWER
I20260807 06:53:44.490978 22503 consensus_queue.cc:260] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.492329 22285 heartbeater.cc:503] Master 127.27.54.126:37889 was elected leader, sending a full tablet report...
I20260807 06:53:44.493140 22503 ts_tablet_manager.cc:1428] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260807 06:53:44.582926 22477 tablet.cc:2369] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:53:44.590620 22518 raft_consensus.cc:482] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:44.591071 22518 raft_consensus.cc:504] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.592937 22518 leader_election.cc:288] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b770d7cab94576b1375a64fb926a4d (127.27.54.65:33991), 7aab068b57714b6e833efc240406ffe8 (127.27.54.67:44489)
I20260807 06:53:44.598886 22512 raft_consensus.cc:482] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:44.599283 22512 raft_consensus.cc:504] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.600797 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.593269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.27.54.66:37575)
0807 06:53:44.594005 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.594029 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.594220 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.594893 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.594900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.594968 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.595421 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.595435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.596506 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.596514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.598154 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.598166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.599726 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.599733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.599939 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.599980 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.600140 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.600304 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":1,"spinlock_wait_cycles":24448,"thread_start_us":335,"threads_started":1}
I20260807 06:53:44.601012 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.594251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:44311 (local address 127.27.54.67:44489)
0807 06:53:44.594463 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.594469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.594925 (+ 456us) server_negotiation.cc:407] Connection header received
0807 06:53:44.595062 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.595069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.595158 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.595265 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.595951 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.596658 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.598490 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.599750 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.599939 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.600020 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.600617 (+ 597us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.600767 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:53:44.601089 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.594088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:44489 (local address 127.27.54.66:44311)
0807 06:53:44.594810 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.594827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.594951 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.595315 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.595319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.595342 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.595712 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.595720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.596818 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.596823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.598294 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.598307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.600061 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.600070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.600272 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.600301 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.600494 (+ 193us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.600629 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"spinlock_wait_cycles":3968,"thread_start_us":341,"threads_started":1}
I20260807 06:53:44.601648 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.593657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:37575 (local address 127.27.54.65:33991)
0807 06:53:44.593940 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.593947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.594156 (+ 209us) server_negotiation.cc:407] Connection header received
0807 06:53:44.594444 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.594450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.594540 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.594606 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.595596 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.596364 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.598340 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.599489 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.599706 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.599794 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.600204 (+ 410us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.600355 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 06:53:44.602434 22410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60f555a4f644ab0af2694a451373712" candidate_uuid: "773c6024ef844e439d5878398f8c47b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aab068b57714b6e833efc240406ffe8" is_pre_election: true
I20260807 06:53:44.602610 22512 leader_election.cc:288] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aab068b57714b6e833efc240406ffe8 (127.27.54.67:44489), 773c6024ef844e439d5878398f8c47b2 (127.27.54.66:41649)
I20260807 06:53:44.603098 22410 raft_consensus.cc:2454] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773c6024ef844e439d5878398f8c47b2 in term 0.
I20260807 06:53:44.603338 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60f555a4f644ab0af2694a451373712" candidate_uuid: "773c6024ef844e439d5878398f8c47b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b770d7cab94576b1375a64fb926a4d" is_pre_election: true
I20260807 06:53:44.603909 22134 leader_election.cc:302] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773c6024ef844e439d5878398f8c47b2, 7aab068b57714b6e833efc240406ffe8; no voters:
I20260807 06:53:44.604110 22040 raft_consensus.cc:2454] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773c6024ef844e439d5878398f8c47b2 in term 0.
I20260807 06:53:44.604420 22518 raft_consensus.cc:2781] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:53:44.604583 22518 raft_consensus.cc:482] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:53:44.604702 22518 raft_consensus.cc:3037] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:44.607834 22518 raft_consensus.cc:504] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.608589 22518 leader_election.cc:288] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [CANDIDATE]: Term 1 election: Requested vote from peers 84b770d7cab94576b1375a64fb926a4d (127.27.54.65:33991), 7aab068b57714b6e833efc240406ffe8 (127.27.54.67:44489)
I20260807 06:53:44.609064 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60f555a4f644ab0af2694a451373712" candidate_uuid: "773c6024ef844e439d5878398f8c47b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b770d7cab94576b1375a64fb926a4d"
I20260807 06:53:44.609311 22040 raft_consensus.cc:3037] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:44.609437 22410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60f555a4f644ab0af2694a451373712" candidate_uuid: "773c6024ef844e439d5878398f8c47b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aab068b57714b6e833efc240406ffe8"
I20260807 06:53:44.609679 22410 raft_consensus.cc:3037] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:44.611807 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.602260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:36705 (local address 127.27.54.67:44489)
0807 06:53:44.602461 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.602468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.604025 (+ 1557us) server_negotiation.cc:407] Connection header received
0807 06:53:44.604449 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.604455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.604539 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.604615 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.605568 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.606344 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.609013 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.610394 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.610549 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.610634 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.611309 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.611477 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260807 06:53:44.612916 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.602888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:44489 (local address 127.27.54.65:36705)
0807 06:53:44.603916 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.603940 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.604157 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.604869 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.604874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.604903 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.605396 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.605412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.606499 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.606506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.608809 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.608825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.610884 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.610892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.611090 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.611130 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.612001 (+ 871us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.612533 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":441,"threads_started":1}
I20260807 06:53:44.613083 22040 raft_consensus.cc:2454] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773c6024ef844e439d5878398f8c47b2 in term 1.
I20260807 06:53:44.613253 22410 raft_consensus.cc:2454] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773c6024ef844e439d5878398f8c47b2 in term 1.
I20260807 06:53:44.613734 22134 leader_election.cc:302] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773c6024ef844e439d5878398f8c47b2, 7aab068b57714b6e833efc240406ffe8; no voters:
I20260807 06:53:44.613976 22410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60f555a4f644ab0af2694a451373712" candidate_uuid: "84b770d7cab94576b1375a64fb926a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aab068b57714b6e833efc240406ffe8" is_pre_election: true
I20260807 06:53:44.614352 22410 raft_consensus.cc:2379] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84b770d7cab94576b1375a64fb926a4d in current term 1: Already voted for candidate 773c6024ef844e439d5878398f8c47b2 in this term.
I20260807 06:53:44.614390 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.604547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:48471 (local address 127.27.54.66:41649)
0807 06:53:44.606861 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.606865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.606879 (+ 14us) server_negotiation.cc:407] Connection header received
0807 06:53:44.606929 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.606932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.606986 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.607089 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:44.607898 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.608842 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.611838 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.613174 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.613331 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.613407 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.614051 (+ 644us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.614191 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2172}
I20260807 06:53:44.615041 22518 raft_consensus.cc:2781] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:44.615116 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.604049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:41649 (local address 127.27.54.65:48471)
0807 06:53:44.605288 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.605305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:44.605401 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:44.607113 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:44.607118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:44.607146 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:44.607647 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.607661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.609014 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.609019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.611600 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:44.611615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.613627 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.613633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:44.613848 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.613884 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:53:44.614582 (+ 698us) client_negotiation.cc:240] Negotiation successful
0807 06:53:44.614753 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":887,"threads_started":1}
I20260807 06:53:44.616093 22518 raft_consensus.cc:686] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 1 LEADER]: Becoming Leader. State: Replica: 773c6024ef844e439d5878398f8c47b2, State: Running, Role: LEADER
I20260807 06:53:44.616562 22228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60f555a4f644ab0af2694a451373712" candidate_uuid: "84b770d7cab94576b1375a64fb926a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773c6024ef844e439d5878398f8c47b2" is_pre_election: true
I20260807 06:53:44.616701 22518 consensus_queue.cc:237] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } }
I20260807 06:53:44.617713 21950 leader_election.cc:302] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84b770d7cab94576b1375a64fb926a4d; no voters: 773c6024ef844e439d5878398f8c47b2, 7aab068b57714b6e833efc240406ffe8
I20260807 06:53:44.618126 22512 raft_consensus.cc:2726] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 06:53:44.621981 21841 catalog_manager.cc:5495] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773c6024ef844e439d5878398f8c47b2 (127.27.54.66). New cstate: current_term: 1 leader_uuid: "773c6024ef844e439d5878398f8c47b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773c6024ef844e439d5878398f8c47b2" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 41649 } health_report { overall_health: HEALTHY } } }
W20260807 06:53:44.635535 22101 tablet.cc:2369] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:53:44.676256 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.670620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.27.54.66:41649)
0807 06:53:44.670848 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.670853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.670873 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:53:44.671039 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.671043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.671124 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.671259 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:53:44.671648 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.672408 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.673873 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.674317 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.674416 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.674997 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:53:44.675511 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:53:44.675650 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.675849 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.675893 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 06:53:44.685250 22410 raft_consensus.cc:1261] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 1 FOLLOWER]: Refusing update from remote peer 773c6024ef844e439d5878398f8c47b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:44.685294 22040 raft_consensus.cc:1261] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 1 FOLLOWER]: Refusing update from remote peer 773c6024ef844e439d5878398f8c47b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:44.686209 22523 consensus_queue.cc:1035] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b770d7cab94576b1375a64fb926a4d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:44.686720 22518 consensus_queue.cc:1035] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aab068b57714b6e833efc240406ffe8" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:44.698174 22528 mvcc.cc:205] Tried to move back new op lower bound from 7315806718700318720 to 7315806718436376576. Current Snapshot: MvccSnapshot[applied={T|T < 7315806718700318720}]
I20260807 06:53:44.699647 22531 mvcc.cc:205] Tried to move back new op lower bound from 7315806718700318720 to 7315806718436376576. Current Snapshot: MvccSnapshot[applied={T|T < 7315806718700318720}]
I20260807 06:53:44.714241 21841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:44.714546 21841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:44.718987 21841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a0d57625560463ab71788433e1477b7: Sending DeleteTablet for 3 replicas of tablet e60f555a4f644ab0af2694a451373712
I20260807 06:53:44.720712 22009 tablet_service.cc:1514] Processing DeleteTablet for tablet e60f555a4f644ab0af2694a451373712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:44 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:47706
I20260807 06:53:44.721019 22385 tablet_service.cc:1514] Processing DeleteTablet for tablet e60f555a4f644ab0af2694a451373712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:44 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:54048
I20260807 06:53:44.721805 22200 tablet_service.cc:1514] Processing DeleteTablet for tablet e60f555a4f644ab0af2694a451373712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:44 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:38182
I20260807 06:53:44.724987 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 21925
I20260807 06:53:44.753182 22543 tablet_replica.cc:330] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: stopping tablet replica
I20260807 06:53:44.753839 22542 tablet_replica.cc:330] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: stopping tablet replica
I20260807 06:53:44.754535 22543 raft_consensus.cc:2229] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:53:44.755149 22542 raft_consensus.cc:2229] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:44.756031 22543 raft_consensus.cc:2258] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:44.756726 22540 tablet_replica.cc:330] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: stopping tablet replica
I20260807 06:53:44.757174 22540 raft_consensus.cc:2229] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 1 FOLLOWER]: Raft consensus shutting down.
Aug 07 06:53:44 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:53:44.757745 22540 raft_consensus.cc:2258] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:44.758520 22542 raft_consensus.cc:2258] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:44.761163 22542 ts_tablet_manager.cc:1905] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:44.761498 22543 ts_tablet_manager.cc:1905] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:44.768110 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.726580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.27.54.65:33991)
0807 06:53:44.726804 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.726811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.731484 (+ 4673us) server_negotiation.cc:407] Connection header received
0807 06:53:44.731848 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.731858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.731969 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.732055 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:44.746936 (+ 14881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.748581 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.751522 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.752488 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.752618 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.759061 (+ 6443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:44.759077 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:44.759094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:44.759138 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:44.763284 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.764405 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.764415 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.764418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.764626 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.765204 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.765209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.765213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.765716 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:44.766037 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.767387 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.767644 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 06:53:44.769032 22095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:44.769586 22543 ts_tablet_manager.cc:1918] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:44.769919 22543 log.cc:1196] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/e60f555a4f644ab0af2694a451373712
I20260807 06:53:44.770641 22543 ts_tablet_manager.cc:1939] T e60f555a4f644ab0af2694a451373712 P 773c6024ef844e439d5878398f8c47b2: Deleting consensus metadata
I20260807 06:53:44.771876 22542 ts_tablet_manager.cc:1918] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:44.772163 22542 log.cc:1196] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/e60f555a4f644ab0af2694a451373712
I20260807 06:53:44.772830 22542 ts_tablet_manager.cc:1939] T e60f555a4f644ab0af2694a451373712 P 7aab068b57714b6e833efc240406ffe8: Deleting consensus metadata
I20260807 06:53:44.772833 21822 catalog_manager.cc:4841] TS 773c6024ef844e439d5878398f8c47b2 (127.27.54.66:41649): tablet e60f555a4f644ab0af2694a451373712 (table test-table [id=7697aceab9044792b8a9f28ad0f155b1]) successfully deleted
I20260807 06:53:44.774874 21822 catalog_manager.cc:4841] TS 7aab068b57714b6e833efc240406ffe8 (127.27.54.67:44489): tablet e60f555a4f644ab0af2694a451373712 (table test-table [id=7697aceab9044792b8a9f28ad0f155b1]) successfully deleted
I20260807 06:53:44.769979 22540 ts_tablet_manager.cc:1905] T e60f555a4f644ab0af2694a451373712 P 84b770d7cab94576b1375a64fb926a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:44.869599 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 21925
I20260807 06:53:44.894529 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 22108
W20260807 06:53:44.894466 21822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.54.65:33991 (error 108)
I20260807 06:53:44.897189 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.896619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47716)
0807 06:53:44.896901 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.897011 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260807 06:53:44.897486 21822 catalog_manager.cc:4584] TS 84b770d7cab94576b1375a64fb926a4d (127.27.54.65:33991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e60f555a4f644ab0af2694a451373712: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Aug 07 06:53:44 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:53:44.913968 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.895471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.27.54.66:41649)
0807 06:53:44.895719 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:44.895725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:44.896009 (+ 284us) server_negotiation.cc:407] Connection header received
0807 06:53:44.896339 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:44.896346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:44.896455 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:44.896580 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:44.898316 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.899322 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.900716 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:44.901341 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:44.901498 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:44.906593 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:44.906615 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:44.906632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:44.906752 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:44.910223 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.911262 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.911274 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.911278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.911451 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:44.911894 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:44.911900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:44.911905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:44.912486 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:44.912856 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:44.913340 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 06:53:44.913605 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:53:44.914752 22280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:44.951500 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.950969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47720)
0807 06:53:44.951207 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.951307 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 06:53:44.952841 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:44.952422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47730)
0807 06:53:44.952599 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:44.952693 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 06:53:45.013851 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.013258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47746)
0807 06:53:45.013556 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:45.013662 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260807 06:53:45.015139 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.014761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47756)
0807 06:53:45.014934 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:45.015006 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 06:53:45.035975 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 22108
I20260807 06:53:45.060216 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 22295
Aug 07 06:53:45 dist-test-slave-3kk6 krb5kdc[21779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:53:45.077562 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.061198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.27.54.67:44489)
0807 06:53:45.061419 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:45.061425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:45.061447 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:53:45.061681 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:45.061685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:45.061779 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:45.061922 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:45.063487 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:45.064190 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:45.065531 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:45.065964 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:45.066108 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:45.071105 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:45.071124 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:45.071142 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:45.071191 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:45.074065 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:45.075193 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:45.075202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:45.075205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:45.075364 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:45.075823 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:45.075828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:45.075830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:45.076235 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:45.076584 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:45.076936 (+ 352us) server_negotiation.cc:299] Negotiation successful
0807 06:53:45.077221 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 06:53:45.078258 22465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:45.096561 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.095999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47768)
0807 06:53:45.096220 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:45.096324 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 06:53:45.098392 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.097589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47780)
0807 06:53:45.098027 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:45.098150 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267}
I20260807 06:53:45.185128 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 22295
I20260807 06:53:45.210872 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 21799
I20260807 06:53:45.212136 21885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:45.326855 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 21799
I20260807 06:53:45.326917 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.326234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33991 (local address 127.0.0.1:47792)
0807 06:53:45.326548 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:45.326731 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:33991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
2026-08-07T06:53:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5073 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T06:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:45Z Disabled control of system clock
I20260807 06:53:45.439396 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:46187
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:46187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:53:45.826591 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:45.827064 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:45.827248 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:45.835227 22558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:45.835376 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:45.835464 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:45.835564 22558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:45.835649 22558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:45.845810 22558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:46187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:45.847633 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:45.850255 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:45.867134 22564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:45.867101 22569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:45.867201 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:45.868129 22558 server_base.cc:1034] running on GCE node
I20260807 06:53:45.869109 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:45.870826 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:45.872085 22558 hybrid_clock.cc:648] HybridClock initialized: now 1786085625872031 us; error 71 us; skew 500 ppm
I20260807 06:53:45.872490 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:45.875101 22558 webserver.cc:460] Webserver started at http://127.27.54.126:45879/ using document root <none> and password file <none>
I20260807 06:53:45.875859 22558 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:45.876019 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:45.876367 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:45.879797 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "2cf46b6b90cf4011952fa124227f3cc1"
format_stamp: "Formatted at 2026-08-07 06:53:45 on dist-test-slave-3kk6"
server_key: "596752cdc9e84a392bc74f87d1f56d77"
server_key_iv: "a21b09b91b4dabdf8269c013be8991df"
server_key_version: "encryptionkey@0"
I20260807 06:53:45.880724 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "2cf46b6b90cf4011952fa124227f3cc1"
format_stamp: "Formatted at 2026-08-07 06:53:45 on dist-test-slave-3kk6"
server_key: "596752cdc9e84a392bc74f87d1f56d77"
server_key_iv: "a21b09b91b4dabdf8269c013be8991df"
server_key_version: "encryptionkey@0"
I20260807 06:53:45.886571 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:53:45.890627 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:45.892292 22558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:45.892522 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "2cf46b6b90cf4011952fa124227f3cc1"
format_stamp: "Formatted at 2026-08-07 06:53:45 on dist-test-slave-3kk6"
server_key: "596752cdc9e84a392bc74f87d1f56d77"
server_key_iv: "a21b09b91b4dabdf8269c013be8991df"
server_key_version: "encryptionkey@0"
I20260807 06:53:45.892791 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:45.914275 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:45.915328 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:45.915686 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:45.936720 22558 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:46187
I20260807 06:53:45.936735 22658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:46187 every 8 connection(s)
I20260807 06:53:45.938259 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:45.939806 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 22558
I20260807 06:53:45.940002 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:45.940608 27865 external_mini_cluster.cc:1428] Setting key 734d78e7e3c2601301ed65adfbdf475d
I20260807 06:53:45.943363 22660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:45.955351 22660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: Bootstrap starting.
I20260807 06:53:45.955493 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:45.942742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.27.54.126:46187)
0807 06:53:45.943960 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:45.943980 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:45.944026 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:53:45.945734 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:45.945757 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:45.946248 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:45.946752 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:45.947378 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:45.948803 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:45.950428 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:45.951201 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:45.951438 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:45.952817 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:45.952825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:45.952841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:45.953490 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:45.953575 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:45.953599 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:45.953738 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:45.953993 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 06:53:45.954342 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":491,"threads_started":1}
I20260807 06:53:45.958812 22660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:45.959880 22660 log.cc:824] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:45.963151 22660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: No bootstrap required, opened a new log
I20260807 06:53:45.971035 22660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf46b6b90cf4011952fa124227f3cc1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46187 } }
I20260807 06:53:45.971467 22660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:45.971577 22660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf46b6b90cf4011952fa124227f3cc1, State: Initialized, Role: FOLLOWER
I20260807 06:53:45.972246 22660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf46b6b90cf4011952fa124227f3cc1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46187 } }
I20260807 06:53:45.972492 22660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:45.972643 22660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:45.972818 22660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:45.975580 22660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf46b6b90cf4011952fa124227f3cc1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46187 } }
I20260807 06:53:45.976248 22660 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf46b6b90cf4011952fa124227f3cc1; no voters:
I20260807 06:53:45.976814 22660 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:45.977002 22667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:45.977752 22667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 2cf46b6b90cf4011952fa124227f3cc1, State: Running, Role: LEADER
I20260807 06:53:45.978345 22667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf46b6b90cf4011952fa124227f3cc1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46187 } }
I20260807 06:53:45.979040 22660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:45.981384 22669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf46b6b90cf4011952fa124227f3cc1. Latest consensus state: current_term: 1 leader_uuid: "2cf46b6b90cf4011952fa124227f3cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf46b6b90cf4011952fa124227f3cc1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46187 } } }
I20260807 06:53:45.981802 22669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:45.987545 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf46b6b90cf4011952fa124227f3cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf46b6b90cf4011952fa124227f3cc1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 46187 } } }
I20260807 06:53:45.987900 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1 [sys.catalog]: This master's current role is: LEADER
W20260807 06:53:45.991623 22689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:53:45.991798 22689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:53:45.991935 22679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:45.993613 22679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:46.002223 22679 catalog_manager.cc:1347] Generated new cluster ID: 27aeb2c9847e4293b96155308e7a7cf4
I20260807 06:53:46.002380 22679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:46.017908 22679 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:46.019380 22679 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:46.029610 22679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: Generated new TSK 0
I20260807 06:53:46.030540 22679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:53:46.043674 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46187
--builtin_ntp_servers=127.27.54.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:53:46.413939 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:46.414265 22695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:46.414366 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:46.421844 22695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:46.421976 22695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:46.422088 22695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:46.432214 22695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:46.433573 22695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:46.435457 22695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:46.449064 22701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:46.449103 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:46.449155 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:46.450464 22695 server_base.cc:1034] running on GCE node
I20260807 06:53:46.451161 22695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:46.452138 22695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:46.453389 22695 hybrid_clock.cc:648] HybridClock initialized: now 1786085626453349 us; error 48 us; skew 500 ppm
I20260807 06:53:46.453742 22695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:46.456377 22695 webserver.cc:460] Webserver started at http://127.27.54.65:39821/ using document root <none> and password file <none>
I20260807 06:53:46.457141 22695 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:46.457288 22695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:46.457634 22695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:46.460950 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "f1285b4c93844f5780f40076332eacda"
format_stamp: "Formatted at 2026-08-07 06:53:46 on dist-test-slave-3kk6"
server_key: "00d58efb21034723ec3aaa8daf67b16f"
server_key_iv: "dcd03edb612cd9c458d5d57b915bcc1b"
server_key_version: "encryptionkey@0"
I20260807 06:53:46.461970 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "f1285b4c93844f5780f40076332eacda"
format_stamp: "Formatted at 2026-08-07 06:53:46 on dist-test-slave-3kk6"
server_key: "00d58efb21034723ec3aaa8daf67b16f"
server_key_iv: "dcd03edb612cd9c458d5d57b915bcc1b"
server_key_version: "encryptionkey@0"
I20260807 06:53:46.467998 22695 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 06:53:46.471855 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:46.473330 22695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:53:46.473484 22695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "f1285b4c93844f5780f40076332eacda"
format_stamp: "Formatted at 2026-08-07 06:53:46 on dist-test-slave-3kk6"
server_key: "00d58efb21034723ec3aaa8daf67b16f"
server_key_iv: "dcd03edb612cd9c458d5d57b915bcc1b"
server_key_version: "encryptionkey@0"
I20260807 06:53:46.473693 22695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:46.499322 22695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:46.500236 22695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:46.500540 22695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:46.501559 22695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:46.503197 22695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:46.503294 22695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:46.503384 22695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:46.503448 22695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:46.539757 22695 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:44643
I20260807 06:53:46.539870 22867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:44643 every 8 connection(s)
I20260807 06:53:46.541208 22695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:46.551543 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 22695
I20260807 06:53:46.551756 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:53:46.552232 27865 external_mini_cluster.cc:1428] Setting key 2affa4d10b296d09c61080a7854d9b45
I20260807 06:53:46.554628 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:46.544015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:46235 (local address 127.27.54.126:46187)
0807 06:53:46.544508 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:46.544513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:46.545614 (+ 1101us) server_negotiation.cc:407] Connection header received
0807 06:53:46.547593 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:46.547599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:46.547691 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:46.547818 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:46.548925 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:46.549672 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:46.551382 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:46.551906 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:46.552041 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:46.553684 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:46.553690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:46.553693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:46.553711 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:46.553753 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:46.553765 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:46.553846 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:46.554172 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 06:53:46.554261 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260807 06:53:46.555476 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46187
--builtin_ntp_servers=127.27.54.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:53:46.555836 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:46.544536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46187 (local address 127.27.54.65:46235)
0807 06:53:46.545443 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:46.545481 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:46.547281 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:46.548059 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:46.548067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:46.548163 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:46.548739 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:46.548755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:46.549847 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:46.549852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:46.551211 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:46.551225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:46.552169 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:46.552180 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:46.552462 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:46.553292 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:46.553298 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:46.553358 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:46.553475 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:46.553890 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:46.553901 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:46.553909 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 06:53:46.554146 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 06:53:46.554568 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
I20260807 06:53:46.557524 22868 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46187
I20260807 06:53:46.557884 22868 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:46.558723 22868 heartbeater.cc:511] Master 127.27.54.126:46187 requested a full tablet report, sending...
I20260807 06:53:46.560884 22608 ts_manager.cc:194] Registered new tserver with Master: f1285b4c93844f5780f40076332eacda (127.27.54.65:44643)
I20260807 06:53:46.562399 22608 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.65:46235
W20260807 06:53:46.926731 22873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:46.927174 22873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:46.927366 22873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:46.935077 22873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:46.935279 22873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:46.935472 22873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:46.945272 22873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:46.946795 22873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:46.948769 22873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:46.963390 22884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:46.963634 22881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:46.963768 22880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:46.964412 22873 server_base.cc:1034] running on GCE node
I20260807 06:53:46.965140 22873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:46.966284 22873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:46.967546 22873 hybrid_clock.cc:648] HybridClock initialized: now 1786085626967481 us; error 65 us; skew 500 ppm
I20260807 06:53:46.967927 22873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:46.970858 22873 webserver.cc:460] Webserver started at http://127.27.54.66:34545/ using document root <none> and password file <none>
I20260807 06:53:46.971735 22873 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:46.971889 22873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:46.972241 22873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:46.975518 22873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "9051289b42c1492ba4bf7c450e320267"
format_stamp: "Formatted at 2026-08-07 06:53:46 on dist-test-slave-3kk6"
server_key: "9e5ac60753c802a6492650eb08020060"
server_key_iv: "ff2d11ab16e2bc18333176e2dadaf489"
server_key_version: "encryptionkey@0"
I20260807 06:53:46.976438 22873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "9051289b42c1492ba4bf7c450e320267"
format_stamp: "Formatted at 2026-08-07 06:53:46 on dist-test-slave-3kk6"
server_key: "9e5ac60753c802a6492650eb08020060"
server_key_iv: "ff2d11ab16e2bc18333176e2dadaf489"
server_key_version: "encryptionkey@0"
I20260807 06:53:46.982390 22873 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 06:53:46.986764 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:46.988773 22873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 06:53:46.988983 22873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "9051289b42c1492ba4bf7c450e320267"
format_stamp: "Formatted at 2026-08-07 06:53:46 on dist-test-slave-3kk6"
server_key: "9e5ac60753c802a6492650eb08020060"
server_key_iv: "ff2d11ab16e2bc18333176e2dadaf489"
server_key_version: "encryptionkey@0"
I20260807 06:53:46.989243 22873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:47.010089 22873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:47.011191 22873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:47.011545 22873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:47.012715 22873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:47.014497 22873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:47.014635 22873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:47.014832 22873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:47.014926 22873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:47.050609 22873 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:45447
I20260807 06:53:47.050812 23058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:45447 every 8 connection(s)
I20260807 06:53:47.052073 22873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:47.055039 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 22873
I20260807 06:53:47.055241 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:53:47.055739 27865 external_mini_cluster.cc:1428] Setting key b470ec2d79e2288c630c7ac122282a4a
I20260807 06:53:47.059221 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46187
--builtin_ntp_servers=127.27.54.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:53:47.067837 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.054860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:46061 (local address 127.27.54.126:46187)
0807 06:53:47.055081 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.055087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.056876 (+ 1789us) server_negotiation.cc:407] Connection header received
0807 06:53:47.059211 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.059215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.059282 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.059418 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:47.060958 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.061727 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.064340 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.064796 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.064898 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.066850 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:47.066858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:47.066866 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:47.066895 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:47.066942 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:47.066961 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:47.067060 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.067367 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.067491 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:53:47.069362 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.055518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46187 (local address 127.27.54.66:46061)
0807 06:53:47.056684 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.056734 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.058887 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.059703 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.059718 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.059844 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:47.060748 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.060778 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.061897 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.061906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.064157 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.064177 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.065053 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.065063 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.065435 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.066395 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:47.066401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:47.066464 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:47.066590 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:47.067078 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:47.067093 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:47.067106 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.067312 (+ 206us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.067800 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":414,"threads_started":1}
I20260807 06:53:47.071007 23059 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46187
I20260807 06:53:47.071385 23059 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:47.072441 23059 heartbeater.cc:511] Master 127.27.54.126:46187 requested a full tablet report, sending...
I20260807 06:53:47.074138 22608 ts_manager.cc:194] Registered new tserver with Master: 9051289b42c1492ba4bf7c450e320267 (127.27.54.66:45447)
I20260807 06:53:47.075093 22608 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.66:46061
W20260807 06:53:47.435252 23064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:47.435595 23064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:47.435766 23064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:47.443478 23064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:47.443610 23064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:47.443804 23064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:47.453830 23064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:46187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:47.455358 23064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:47.457291 23064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:47.472075 23077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:47.472122 23073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:47.472023 23074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:47.473189 23064 server_base.cc:1034] running on GCE node
I20260807 06:53:47.473968 23064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:47.475149 23064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:47.476395 23064 hybrid_clock.cc:648] HybridClock initialized: now 1786085627476329 us; error 68 us; skew 500 ppm
I20260807 06:53:47.476745 23064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:47.479547 23064 webserver.cc:460] Webserver started at http://127.27.54.67:46389/ using document root <none> and password file <none>
I20260807 06:53:47.480369 23064 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:47.480530 23064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:47.480906 23064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:47.484503 23064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "ea24cf3fdea847ee99d4da71a5087325"
format_stamp: "Formatted at 2026-08-07 06:53:47 on dist-test-slave-3kk6"
server_key: "7629379927d15af4420098c37e98ddc6"
server_key_iv: "7dac10d51e5da018dc9828fc75628503"
server_key_version: "encryptionkey@0"
I20260807 06:53:47.485471 23064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "ea24cf3fdea847ee99d4da71a5087325"
format_stamp: "Formatted at 2026-08-07 06:53:47 on dist-test-slave-3kk6"
server_key: "7629379927d15af4420098c37e98ddc6"
server_key_iv: "7dac10d51e5da018dc9828fc75628503"
server_key_version: "encryptionkey@0"
I20260807 06:53:47.491245 23064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 06:53:47.495306 23085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:47.496961 23064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 06:53:47.497156 23064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "ea24cf3fdea847ee99d4da71a5087325"
format_stamp: "Formatted at 2026-08-07 06:53:47 on dist-test-slave-3kk6"
server_key: "7629379927d15af4420098c37e98ddc6"
server_key_iv: "7dac10d51e5da018dc9828fc75628503"
server_key_version: "encryptionkey@0"
I20260807 06:53:47.497406 23064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:47.527666 23064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:47.528645 23064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:47.528995 23064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:47.530088 23064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:47.531878 23064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:47.532008 23064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:47.532157 23064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:47.532249 23064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:47.566709 22868 heartbeater.cc:503] Master 127.27.54.126:46187 was elected leader, sending a full tablet report...
I20260807 06:53:47.569321 23064 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:39585
I20260807 06:53:47.569393 23250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:39585 every 8 connection(s)
I20260807 06:53:47.570885 23064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:47.580515 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 23064
I20260807 06:53:47.580718 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:53:47.581315 27865 external_mini_cluster.cc:1428] Setting key 5c031db30dfb70de682ab2e954b2f7ec
I20260807 06:53:47.585297 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.573492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:33951 (local address 127.27.54.126:46187)
0807 06:53:47.573962 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.573967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.575226 (+ 1259us) server_negotiation.cc:407] Connection header received
0807 06:53:47.577239 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.577245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.577343 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.577473 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:47.578679 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.579548 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.581966 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.582410 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.582559 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.584337 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:47.584343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:47.584346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:47.584365 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:47.584433 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:47.584452 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:47.584554 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.584862 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.584975 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":272,"threads_started":1}
I20260807 06:53:47.586409 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.573989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:46187 (local address 127.27.54.67:33951)
0807 06:53:47.575041 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.575081 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.576891 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.577755 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.577765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.577861 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:47.578463 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.578486 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.579723 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.579727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.581786 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.581808 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.582720 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.582730 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.583067 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.583948 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:47.583952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:47.584020 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:47.584136 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:47.584588 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:47.584599 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:47.584608 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.584835 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.585184 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":324,"threads_started":1}
I20260807 06:53:47.587970 23251 heartbeater.cc:347] Connected to a master server at 127.27.54.126:46187
I20260807 06:53:47.588310 23251 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:47.589129 23251 heartbeater.cc:511] Master 127.27.54.126:46187 requested a full tablet report, sending...
I20260807 06:53:47.590758 22608 ts_manager.cc:194] Registered new tserver with Master: ea24cf3fdea847ee99d4da71a5087325 (127.27.54.67:39585)
I20260807 06:53:47.591614 22608 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.67:33951
I20260807 06:53:47.597752 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:53:47.609056 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.602563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.27.54.126:46187)
0807 06:53:47.602844 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.602851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.603194 (+ 343us) server_negotiation.cc:407] Connection header received
0807 06:53:47.603362 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.603367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.603438 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.603529 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:47.604034 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.604782 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.606267 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.606729 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.606840 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.608028 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:47.608033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:47.608037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:47.608056 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:47.608128 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:47.608153 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:47.608387 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.608614 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.608690 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:53:47.613677 22608 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:53:47.621063 22608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:53:47.648923 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.634993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:45447 (local address 127.0.0.1:54998)
0807 06:53:47.636030 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.636085 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.636342 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.637979 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.637991 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.638038 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:47.638629 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.638673 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.641393 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.641403 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.644895 (+ 3492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.644922 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.647778 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.647786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.648065 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.648161 (+ 96us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.648305 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.648543 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":1528,"threads_started":1}
I20260807 06:53:47.650133 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.635083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.27.54.66:45447)
0807 06:53:47.637156 (+ 2073us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.637175 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.637210 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:53:47.637319 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.637329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.637606 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.638974 (+ 1368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:47.639287 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.641160 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.645280 (+ 4120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.647466 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.648733 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.649053 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.649157 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.649611 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1907,"thread_start_us":1789,"threads_started":1}
I20260807 06:53:47.654708 22963 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed6d13725442f68fe08fdbfb94c68d (DEFAULT_TABLE table=test-table [id=1016c3a333f34de8bec1d7cb9c3d03ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:47.656313 22963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed6d13725442f68fe08fdbfb94c68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:47.656558 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.635529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:39585 (local address 127.0.0.1:60884)
0807 06:53:47.647794 (+ 12265us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.647822 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.648026 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.649524 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.649531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.649557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:47.649933 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.649949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.652037 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.652041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.653448 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.653462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.655699 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.655706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.655867 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.655901 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.655988 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.656160 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12086,"thread_start_us":12625,"threads_started":1}
I20260807 06:53:47.657440 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.636178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.27.54.67:39585)
0807 06:53:47.647735 (+ 11557us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.647744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.648479 (+ 735us) server_negotiation.cc:407] Connection header received
0807 06:53:47.648620 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.648628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.648927 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.649180 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:47.650171 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.651868 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.653674 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.655236 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.655387 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.656957 (+ 1570us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.657030 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.657166 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11324,"thread_start_us":11559,"threads_started":1}
I20260807 06:53:47.659264 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.637561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:38076)
0807 06:53:47.648431 (+ 10870us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.648448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.648561 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.651268 (+ 2707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.651273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.651304 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:47.651644 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.651655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.653937 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.653943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.656135 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.656148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.658328 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.658335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.658528 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.658565 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.658779 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.658938 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10731,"thread_start_us":9888,"threads_started":1}
I20260807 06:53:47.659440 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.638346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.27.54.65:44643)
0807 06:53:47.650536 (+ 12190us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.650545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.650577 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:53:47.650722 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.650732 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.651031 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.651244 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:47.652262 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.653777 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.656380 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.658083 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.658251 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.658550 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.658857 (+ 307us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.659032 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11973,"thread_start_us":11885,"threads_started":1}
I20260807 06:53:47.661746 23163 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed6d13725442f68fe08fdbfb94c68d (DEFAULT_TABLE table=test-table [id=1016c3a333f34de8bec1d7cb9c3d03ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:47.663563 23163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed6d13725442f68fe08fdbfb94c68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:47.663730 22781 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed6d13725442f68fe08fdbfb94c68d (DEFAULT_TABLE table=test-table [id=1016c3a333f34de8bec1d7cb9c3d03ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:47.665375 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed6d13725442f68fe08fdbfb94c68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:47.670632 23276 tablet_bootstrap.cc:492] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Bootstrap starting.
I20260807 06:53:47.674149 23276 tablet_bootstrap.cc:654] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:47.675397 23276 log.cc:824] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:47.679137 23276 tablet_bootstrap.cc:492] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: No bootstrap required, opened a new log
I20260807 06:53:47.679481 23276 ts_tablet_manager.cc:1397] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 06:53:47.680764 23278 tablet_bootstrap.cc:492] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Bootstrap starting.
I20260807 06:53:47.682890 23279 tablet_bootstrap.cc:492] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Bootstrap starting.
I20260807 06:53:47.685166 23278 tablet_bootstrap.cc:654] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:47.686233 23279 tablet_bootstrap.cc:654] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:47.686578 23278 log.cc:824] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:47.687520 23279 log.cc:824] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:47.690413 23278 tablet_bootstrap.cc:492] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: No bootstrap required, opened a new log
I20260807 06:53:47.690783 23278 ts_tablet_manager.cc:1397] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260807 06:53:47.690939 23279 tablet_bootstrap.cc:492] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: No bootstrap required, opened a new log
I20260807 06:53:47.691218 23279 ts_tablet_manager.cc:1397] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 06:53:47.690948 23276 raft_consensus.cc:348] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.691468 23276 raft_consensus.cc:374] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:47.691604 23276 raft_consensus.cc:729] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9051289b42c1492ba4bf7c450e320267, State: Initialized, Role: FOLLOWER
I20260807 06:53:47.692303 23276 consensus_queue.cc:260] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.693562 23059 heartbeater.cc:503] Master 127.27.54.126:46187 was elected leader, sending a full tablet report...
I20260807 06:53:47.694283 23276 ts_tablet_manager.cc:1428] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260807 06:53:47.702617 23279 raft_consensus.cc:348] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.702617 23278 raft_consensus.cc:348] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.703240 23278 raft_consensus.cc:374] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:47.703239 23279 raft_consensus.cc:374] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:47.703356 23279 raft_consensus.cc:729] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1285b4c93844f5780f40076332eacda, State: Initialized, Role: FOLLOWER
I20260807 06:53:47.703356 23278 raft_consensus.cc:729] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea24cf3fdea847ee99d4da71a5087325, State: Initialized, Role: FOLLOWER
I20260807 06:53:47.704094 23279 consensus_queue.cc:260] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.704101 23278 consensus_queue.cc:260] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.705334 23251 heartbeater.cc:503] Master 127.27.54.126:46187 was elected leader, sending a full tablet report...
I20260807 06:53:47.706290 23278 ts_tablet_manager.cc:1428] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260807 06:53:47.706447 23279 ts_tablet_manager.cc:1428] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260807 06:53:47.793970 22869 tablet.cc:2369] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:53:47.804396 23060 tablet.cc:2369] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 06:53:47.822950 23252 tablet.cc:2369] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:53:47.973240 23286 raft_consensus.cc:482] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:47.973603 23286 raft_consensus.cc:504] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.975636 23286 leader_election.cc:288] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1285b4c93844f5780f40076332eacda (127.27.54.65:44643), ea24cf3fdea847ee99d4da71a5087325 (127.27.54.67:39585)
I20260807 06:53:47.983466 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.976280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:48475 (local address 127.27.54.67:39585)
0807 06:53:47.976548 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.976556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.977591 (+ 1035us) server_negotiation.cc:407] Connection header received
0807 06:53:47.977852 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.977857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.977917 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.978022 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:47.978823 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.979651 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.981625 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.982429 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.982645 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.982761 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.983078 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.983219 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 06:53:47.983497 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.976039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:58285 (local address 127.27.54.65:44643)
0807 06:53:47.976302 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:47.976312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:47.977513 (+ 1201us) server_negotiation.cc:407] Connection header received
0807 06:53:47.977805 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:47.977809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:47.977884 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:47.977989 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:47.978707 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.979594 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.981695 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.982506 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.982722 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.982807 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:47.983094 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 06:53:47.983237 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 06:53:47.983642 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.976031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:39585 (local address 127.27.54.66:48475)
0807 06:53:47.977492 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.977508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.977672 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.978035 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.978040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.978063 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:47.978598 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.978609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.979789 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.979798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.981411 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.981427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.982635 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.982644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.982859 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.982888 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.982995 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.983197 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"mutex_wait_us":62,"thread_start_us":691,"threads_started":1}
I20260807 06:53:47.983675 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:47.975903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.27.54.66:58285)
0807 06:53:47.977364 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:47.977397 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:47.977601 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:47.978010 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:47.978014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:47.978048 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:47.978530 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.978545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.979713 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.979720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.981434 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:47.981447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:47.982729 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:47.982736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:47.982927 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:47.982949 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 06:53:47.983063 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 06:53:47.983253 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"mutex_wait_us":70,"thread_start_us":1195,"threads_started":1}
I20260807 06:53:47.984966 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed6d13725442f68fe08fdbfb94c68d" candidate_uuid: "9051289b42c1492ba4bf7c450e320267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea24cf3fdea847ee99d4da71a5087325" is_pre_election: true
I20260807 06:53:47.984972 22812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed6d13725442f68fe08fdbfb94c68d" candidate_uuid: "9051289b42c1492ba4bf7c450e320267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1285b4c93844f5780f40076332eacda" is_pre_election: true
I20260807 06:53:47.985528 22812 raft_consensus.cc:2454] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9051289b42c1492ba4bf7c450e320267 in term 0.
I20260807 06:53:47.985528 23187 raft_consensus.cc:2454] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9051289b42c1492ba4bf7c450e320267 in term 0.
I20260807 06:53:47.986236 22900 leader_election.cc:302] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9051289b42c1492ba4bf7c450e320267, ea24cf3fdea847ee99d4da71a5087325; no voters:
I20260807 06:53:47.986707 23286 raft_consensus.cc:2781] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:53:47.986852 23286 raft_consensus.cc:482] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:53:47.986937 23286 raft_consensus.cc:3037] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:47.989235 23286 raft_consensus.cc:504] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:47.989943 23286 leader_election.cc:288] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [CANDIDATE]: Term 1 election: Requested vote from peers f1285b4c93844f5780f40076332eacda (127.27.54.65:44643), ea24cf3fdea847ee99d4da71a5087325 (127.27.54.67:39585)
I20260807 06:53:47.990413 22812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed6d13725442f68fe08fdbfb94c68d" candidate_uuid: "9051289b42c1492ba4bf7c450e320267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1285b4c93844f5780f40076332eacda"
I20260807 06:53:47.990490 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed6d13725442f68fe08fdbfb94c68d" candidate_uuid: "9051289b42c1492ba4bf7c450e320267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea24cf3fdea847ee99d4da71a5087325"
I20260807 06:53:47.990702 22812 raft_consensus.cc:3037] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:47.990720 23187 raft_consensus.cc:3037] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:47.994134 22812 raft_consensus.cc:2454] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9051289b42c1492ba4bf7c450e320267 in term 1.
I20260807 06:53:47.994134 23187 raft_consensus.cc:2454] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9051289b42c1492ba4bf7c450e320267 in term 1.
I20260807 06:53:47.994665 22898 leader_election.cc:302] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9051289b42c1492ba4bf7c450e320267, f1285b4c93844f5780f40076332eacda; no voters:
I20260807 06:53:47.995164 23286 raft_consensus.cc:2781] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:47.995692 23286 raft_consensus.cc:686] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 1 LEADER]: Becoming Leader. State: Replica: 9051289b42c1492ba4bf7c450e320267, State: Running, Role: LEADER
I20260807 06:53:47.996225 23286 consensus_queue.cc:237] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } }
I20260807 06:53:48.001787 22607 catalog_manager.cc:5495] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9051289b42c1492ba4bf7c450e320267 (127.27.54.66). New cstate: current_term: 1 leader_uuid: "9051289b42c1492ba4bf7c450e320267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9051289b42c1492ba4bf7c450e320267" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 45447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 } health_report { overall_health: UNKNOWN } } }
I20260807 06:53:48.022521 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.016701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.27.54.66:45447)
0807 06:53:48.016893 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:48.016898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:48.016916 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:53:48.017019 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:48.017023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:48.017123 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:48.017280 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:53:48.017814 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.018759 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.020225 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.020704 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.020828 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:48.021357 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:53:48.021920 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:53:48.022072 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:48.022226 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 06:53:48.022271 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:53:48.031832 22812 raft_consensus.cc:1261] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 1 FOLLOWER]: Refusing update from remote peer 9051289b42c1492ba4bf7c450e320267: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:48.031832 23187 raft_consensus.cc:1261] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 1 FOLLOWER]: Refusing update from remote peer 9051289b42c1492ba4bf7c450e320267: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:48.033020 23293 consensus_queue.cc:1035] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1285b4c93844f5780f40076332eacda" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:48.033418 23286 consensus_queue.cc:1035] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea24cf3fdea847ee99d4da71a5087325" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:48.051203 23297 mvcc.cc:205] Tried to move back new op lower bound from 7315806732407205888 to 7315806732277166080. Current Snapshot: MvccSnapshot[applied={T|T < 7315806732407205888}]
I20260807 06:53:48.058967 23296 mvcc.cc:205] Tried to move back new op lower bound from 7315806732407205888 to 7315806732277166080. Current Snapshot: MvccSnapshot[applied={T|T < 7315806732407205888 or (T in {7315806732407205888})}]
I20260807 06:53:48.068188 22607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:48.068498 22607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:48.072476 22607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cf46b6b90cf4011952fa124227f3cc1: Sending DeleteTablet for 3 replicas of tablet a2ed6d13725442f68fe08fdbfb94c68d
I20260807 06:53:48.073949 22963 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ed6d13725442f68fe08fdbfb94c68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:48 UTC) from {username='slave'} at 127.0.0.1:54998
I20260807 06:53:48.074260 22781 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ed6d13725442f68fe08fdbfb94c68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:48 UTC) from {username='slave'} at 127.0.0.1:38076
I20260807 06:53:48.076574 23163 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ed6d13725442f68fe08fdbfb94c68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:48 UTC) from {username='slave'} at 127.0.0.1:60884
I20260807 06:53:48.081166 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 22695
I20260807 06:53:48.092361 23318 tablet_replica.cc:330] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: stopping tablet replica
I20260807 06:53:48.093422 23317 tablet_replica.cc:330] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: stopping tablet replica
I20260807 06:53:48.093549 23318 raft_consensus.cc:2229] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:48.094210 23318 raft_consensus.cc:2258] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:48.095115 23316 tablet_replica.cc:330] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: stopping tablet replica
I20260807 06:53:48.095618 23316 raft_consensus.cc:2229] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:53:48.095796 23317 raft_consensus.cc:2229] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:48.096366 23316 raft_consensus.cc:2258] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:48.097476 23318 ts_tablet_manager.cc:1905] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:48.100183 23317 raft_consensus.cc:2258] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:48.104400 23316 ts_tablet_manager.cc:1905] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:48.105337 23317 ts_tablet_manager.cc:1905] T a2ed6d13725442f68fe08fdbfb94c68d P f1285b4c93844f5780f40076332eacda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:48.107213 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.083462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.27.54.65:44643)
0807 06:53:48.083764 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:48.083773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:48.091654 (+ 7881us) server_negotiation.cc:407] Connection header received
0807 06:53:48.094275 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:48.094286 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:48.094441 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:48.094544 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:48.096528 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.097827 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.101072 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.101762 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.101885 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:48.105185 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:48.105201 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:48.105223 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:48.106056 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:48.106127 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:48.106163 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:48.106288 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:48.106612 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 06:53:48.106812 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260807 06:53:48.108736 22859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:48.113270 23316 ts_tablet_manager.cc:1918] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:48.113514 23316 log.cc:1196] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/a2ed6d13725442f68fe08fdbfb94c68d
I20260807 06:53:48.114321 23316 ts_tablet_manager.cc:1939] T a2ed6d13725442f68fe08fdbfb94c68d P 9051289b42c1492ba4bf7c450e320267: Deleting consensus metadata
I20260807 06:53:48.116796 23318 ts_tablet_manager.cc:1918] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:48.117008 23318 log.cc:1196] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/a2ed6d13725442f68fe08fdbfb94c68d
I20260807 06:53:48.117018 22582 catalog_manager.cc:4841] TS 9051289b42c1492ba4bf7c450e320267 (127.27.54.66:45447): tablet a2ed6d13725442f68fe08fdbfb94c68d (table test-table [id=1016c3a333f34de8bec1d7cb9c3d03ca]) successfully deleted
I20260807 06:53:48.117623 23318 ts_tablet_manager.cc:1939] T a2ed6d13725442f68fe08fdbfb94c68d P ea24cf3fdea847ee99d4da71a5087325: Deleting consensus metadata
I20260807 06:53:48.119686 22584 catalog_manager.cc:4841] TS ea24cf3fdea847ee99d4da71a5087325 (127.27.54.67:39585): tablet a2ed6d13725442f68fe08fdbfb94c68d (table test-table [id=1016c3a333f34de8bec1d7cb9c3d03ca]) successfully deleted
I20260807 06:53:48.216020 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 22695
I20260807 06:53:48.247604 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 22873
W20260807 06:53:48.247666 22582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.54.65:44643 (error 108)
I20260807 06:53:48.249907 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.248898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33884)
0807 06:53:48.249545 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.249688 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":385,"threads_started":1}
W20260807 06:53:48.250223 22582 catalog_manager.cc:4584] TS f1285b4c93844f5780f40076332eacda (127.27.54.65:44643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2ed6d13725442f68fe08fdbfb94c68d: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
I20260807 06:53:48.256111 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.248916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.27.54.66:45447)
0807 06:53:48.249108 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:48.249114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:48.249137 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:53:48.249326 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:48.249332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:48.249450 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:48.249600 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:48.250245 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.251264 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.252609 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.253089 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.253244 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:48.254496 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:48.254506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:48.254521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:48.255269 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:48.255319 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:48.255341 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:48.255478 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:48.255623 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 06:53:48.255754 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 06:53:48.256960 23054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:48.304062 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.303560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33892)
0807 06:53:48.303797 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.303895 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 06:53:48.305183 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.304787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33900)
0807 06:53:48.304941 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.305035 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 06:53:48.365764 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.365235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33906)
0807 06:53:48.365508 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.365606 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260807 06:53:48.366904 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.366459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33912)
0807 06:53:48.366613 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.366748 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 06:53:48.375631 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 22873
I20260807 06:53:48.407706 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 23064
I20260807 06:53:48.418213 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.408571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.27.54.67:39585)
0807 06:53:48.408741 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:48.408746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:48.408797 (+ 51us) server_negotiation.cc:407] Connection header received
0807 06:53:48.408978 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:48.408983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:48.409068 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:48.409173 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:48.409549 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.410385 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.411831 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:48.412257 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:48.412358 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:48.416744 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:48.416755 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:48.416770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:48.417416 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:48.417468 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:48.417489 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:48.417592 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:48.417825 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 06:53:48.417943 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:53:48.419003 23243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:48.447813 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.447273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33920)
0807 06:53:48.447521 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.447652 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 06:53:48.448995 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.448564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33936)
0807 06:53:48.448765 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.448835 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260807 06:53:48.528694 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 23064
I20260807 06:53:48.553261 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 22558
I20260807 06:53:48.554461 22651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:48.668574 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:48.667771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:44643 (local address 127.0.0.1:33948)
0807 06:53:48.668138 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:48.668320 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.65:44643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260807 06:53:48.668886 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 22558
2026-08-07T06:53:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3340 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:53:48 dist-test-slave-3kk6 krb5kdc[23336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:53:48 dist-test-slave-3kk6 krb5kdc[23336](info): set up 2 sockets
Aug 07 06:53:48 dist-test-slave-3kk6 krb5kdc[23336](info): commencing operation
krb5kdc: starting...
W20260807 06:53:50.033999 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.004s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:53:50 dist-test-slave-3kk6 krb5kdc[23336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:53:50.418063 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34451
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:34451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:50.788151 23354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:50.788506 23354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:50.788674 23354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:50.796419 23354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:50.796564 23354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:50.796648 23354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:50.796754 23354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:50.796844 23354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:50.807528 23354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:34451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:50.809065 23354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:50.810504 23354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:50.823961 23364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:50.824029 23361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:50.823912 23360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:50.825047 23354 server_base.cc:1034] running on GCE node
I20260807 06:53:50.826041 23354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:50.827826 23354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:50.829102 23354 hybrid_clock.cc:648] HybridClock initialized: now 1786085630829062 us; error 66 us; skew 500 ppm
I20260807 06:53:50.829519 23354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:50 dist-test-slave-3kk6 krb5kdc[23336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085630, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:50.837512 23354 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:53:50.839574 23354 webserver.cc:460] Webserver started at http://127.27.54.126:45883/ using document root <none> and password file <none>
I20260807 06:53:50.840382 23354 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:50.840557 23354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:50.840924 23354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:50.844460 23354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "096d8e25f42944e3b61d2f02043ff1a3"
format_stamp: "Formatted at 2026-08-07 06:53:50 on dist-test-slave-3kk6"
server_key: "b10501292c8148e497b7923757b988b0"
server_key_iv: "0ccb2237217087b497d7d7fbbab0a9bb"
server_key_version: "encryptionkey@0"
I20260807 06:53:50.845366 23354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "096d8e25f42944e3b61d2f02043ff1a3"
format_stamp: "Formatted at 2026-08-07 06:53:50 on dist-test-slave-3kk6"
server_key: "b10501292c8148e497b7923757b988b0"
server_key_iv: "0ccb2237217087b497d7d7fbbab0a9bb"
server_key_version: "encryptionkey@0"
I20260807 06:53:50.851414 23354 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260807 06:53:50.855397 23375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:50.857029 23354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:53:50.857196 23354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "096d8e25f42944e3b61d2f02043ff1a3"
format_stamp: "Formatted at 2026-08-07 06:53:50 on dist-test-slave-3kk6"
server_key: "b10501292c8148e497b7923757b988b0"
server_key_iv: "0ccb2237217087b497d7d7fbbab0a9bb"
server_key_version: "encryptionkey@0"
I20260807 06:53:50.857388 23354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:50.889573 23354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:50.894266 23354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:50.894546 23354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:50.915151 23354 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:34451
I20260807 06:53:50.915170 23449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:34451 every 8 connection(s)
I20260807 06:53:50.916883 23354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:50.921103 23450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:50.926601 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 23354
I20260807 06:53:50.926856 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:50.927448 27865 external_mini_cluster.cc:1428] Setting key 9b2f2b0306ab62cebd9db81d7d93a29a
I20260807 06:53:50.933677 23450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3: Bootstrap starting.
I20260807 06:53:50.937150 23450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:50.938230 23450 log.cc:824] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:50.942144 23450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3: No bootstrap required, opened a new log
Aug 07 06:53:50 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:50.950847 23450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096d8e25f42944e3b61d2f02043ff1a3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34451 } }
I20260807 06:53:50.951257 23450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:50.951400 23450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096d8e25f42944e3b61d2f02043ff1a3, State: Initialized, Role: FOLLOWER
I20260807 06:53:50.952167 23450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096d8e25f42944e3b61d2f02043ff1a3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34451 } }
I20260807 06:53:50.952461 23450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:50.952625 23450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:50.952826 23450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:50.953438 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:50.929646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.27.54.126:34451)
0807 06:53:50.930751 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:50.930768 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:50.930823 (+ 55us) server_negotiation.cc:407] Connection header received
0807 06:53:50.932730 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:50.932758 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:50.933209 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:50.933681 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:50.935469 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:50.936832 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:50.938336 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:50.939587 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:50.939813 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:50.945114 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:50.945123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:50.945138 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:50.945176 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:50.948652 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:50.949442 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:50.949449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:50.949456 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:50.949640 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:50.950091 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:50.950096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:50.950100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:50.950758 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:50.951089 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:50.951665 (+ 576us) server_negotiation.cc:299] Negotiation successful
0807 06:53:50.952137 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":435,"threads_started":1}
I20260807 06:53:50.956040 23450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096d8e25f42944e3b61d2f02043ff1a3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34451 } }
I20260807 06:53:50.956689 23450 leader_election.cc:302] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 096d8e25f42944e3b61d2f02043ff1a3; no voters:
I20260807 06:53:50.957342 23450 leader_election.cc:288] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:50.957489 23458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:50.958226 23458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [term 1 LEADER]: Becoming Leader. State: Replica: 096d8e25f42944e3b61d2f02043ff1a3, State: Running, Role: LEADER
I20260807 06:53:50.958851 23458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096d8e25f42944e3b61d2f02043ff1a3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34451 } }
I20260807 06:53:50.959618 23450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:50.961645 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "096d8e25f42944e3b61d2f02043ff1a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096d8e25f42944e3b61d2f02043ff1a3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34451 } } }
I20260807 06:53:50.962074 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:50.964601 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 096d8e25f42944e3b61d2f02043ff1a3. Latest consensus state: current_term: 1 leader_uuid: "096d8e25f42944e3b61d2f02043ff1a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096d8e25f42944e3b61d2f02043ff1a3" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34451 } } }
I20260807 06:53:50.965006 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3 [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:50.966833 23473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:50.972285 23473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:50.981469 23473 catalog_manager.cc:1347] Generated new cluster ID: a997c34d1b1e49d1910ecb99451791d3
I20260807 06:53:50.981618 23473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:51.000705 23473 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:51.002746 23473 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:51.012104 23473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3: Generated new TSK 0
I20260807 06:53:51.013281 23473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.54.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
Entry for principal kudu/127.27.54.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab.
I20260807 06:53:51.107208 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34451
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:51.525906 23501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:51.526320 23501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:51.526501 23501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:51.534319 23501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:51.534518 23501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:51.534789 23501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:51.546049 23501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.65.keytab
--principal=kudu/127.27.54.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:51.547611 23501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:51.549052 23501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:51.564600 23511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:51.564646 23509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:51.564574 23513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:51.565426 23501 server_base.cc:1034] running on GCE node
I20260807 06:53:51.566426 23501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:51.567646 23501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:51.568897 23501 hybrid_clock.cc:648] HybridClock initialized: now 1786085631568822 us; error 72 us; skew 500 ppm
I20260807 06:53:51.569296 23501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:51 dist-test-slave-3kk6 krb5kdc[23336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085631, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:51.573110 23501 init.cc:377] Logged in from keytab as kudu/127.27.54.65@KRBTEST.COM (short username kudu)
I20260807 06:53:51.574990 23501 webserver.cc:460] Webserver started at http://127.27.54.65:41025/ using document root <none> and password file <none>
I20260807 06:53:51.575733 23501 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:51.575847 23501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:51.576133 23501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:51.579315 23501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "ef455e385a214c379cd235a567b23e7b"
format_stamp: "Formatted at 2026-08-07 06:53:51 on dist-test-slave-3kk6"
server_key: "36ebd13e04d84386cf3a9b19a1e0067b"
server_key_iv: "18b9296209252f4e739628a42b26e5c4"
server_key_version: "encryptionkey@0"
I20260807 06:53:51.580129 23501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "ef455e385a214c379cd235a567b23e7b"
format_stamp: "Formatted at 2026-08-07 06:53:51 on dist-test-slave-3kk6"
server_key: "36ebd13e04d84386cf3a9b19a1e0067b"
server_key_iv: "18b9296209252f4e739628a42b26e5c4"
server_key_version: "encryptionkey@0"
I20260807 06:53:51.585944 23501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 06:53:51.589745 23524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:51.591329 23501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:51.591542 23501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "ef455e385a214c379cd235a567b23e7b"
format_stamp: "Formatted at 2026-08-07 06:53:51 on dist-test-slave-3kk6"
server_key: "36ebd13e04d84386cf3a9b19a1e0067b"
server_key_iv: "18b9296209252f4e739628a42b26e5c4"
server_key_version: "encryptionkey@0"
I20260807 06:53:51.591789 23501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:51.606633 23501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:51.611302 23501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:51.611677 23501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:51.612861 23501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:51.614336 23501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:51.614457 23501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:51.614622 23501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:51.614724 23501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 06:53:51.651484 23501 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:32865
I20260807 06:53:51.651531 23674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:32865 every 8 connection(s)
I20260807 06:53:51.652945 23501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:51.654011 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 23501
I20260807 06:53:51.654222 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:53:51.654808 27865 external_mini_cluster.cc:1428] Setting key 1cc1fb142ef269ace510b1338bca2c51
Aug 07 06:53:51 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085631, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.65@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:51.676975 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:51.656048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:60961 (local address 127.27.54.126:34451)
0807 06:53:51.656596 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:51.656601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:51.657924 (+ 1323us) server_negotiation.cc:407] Connection header received
0807 06:53:51.660047 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:51.660056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:51.660156 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:51.660284 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:51.662161 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:51.663353 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:51.665025 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:51.665750 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:51.665917 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:51.669941 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:51.669951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:51.669959 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:51.670008 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:51.673551 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:51.674412 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:51.674418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:51.674422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:51.674561 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:51.675070 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:51.675076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:51.675079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:51.675392 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:51.675558 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:51.676302 (+ 744us) server_negotiation.cc:299] Negotiation successful
0807 06:53:51.676577 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":293,"threads_started":1}
I20260807 06:53:51.678041 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:51.656759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34451 (local address 127.27.54.65:60961)
0807 06:53:51.657721 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:51.657773 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:51.659680 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:51.660517 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:51.660526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:51.661285 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:51.661914 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:51.661933 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:51.663523 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:51.663528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:51.664840 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:51.664853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:51.666088 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:51.666097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:51.666324 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:51.667234 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:51.667259 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:51.669664 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:51.673755 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:51.673765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:51.673791 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:51.674266 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:51.674721 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:51.674726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:51.674730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:51.674945 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:51.675529 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:51.675537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:51.675921 (+ 384us) client_negotiation.cc:769] Sending connection context
0807 06:53:51.676201 (+ 280us) client_negotiation.cc:240] Negotiation successful
0807 06:53:51.676820 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":414,"threads_started":1}
I20260807 06:53:51.679880 23675 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34451
I20260807 06:53:51.680238 23675 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:51.681042 23675 heartbeater.cc:511] Master 127.27.54.126:34451 requested a full tablet report, sending...
I20260807 06:53:51.683420 23406 ts_manager.cc:194] Registered new tserver with Master: ef455e385a214c379cd235a567b23e7b (127.27.54.65:32865)
I20260807 06:53:51.684978 23406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.65@KRBTEST.COM'} at 127.27.54.65:60961
WARNING: no policy specified for kudu/127.27.54.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
Entry for principal kudu/127.27.54.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab.
I20260807 06:53:51.734761 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34451
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:52.101513 23688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:52.101948 23688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:52.102085 23688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:52.111084 23688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:52.111215 23688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:52.111419 23688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:52.121628 23688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.66.keytab
--principal=kudu/127.27.54.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:52.123111 23688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:52.124495 23688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:52.139257 23695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:52.139370 23698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:52.139317 23694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:52.140405 23688 server_base.cc:1034] running on GCE node
I20260807 06:53:52.141053 23688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:52.142022 23688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:52.143229 23688 hybrid_clock.cc:648] HybridClock initialized: now 1786085632143186 us; error 35 us; skew 500 ppm
I20260807 06:53:52.143569 23688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:52 dist-test-slave-3kk6 krb5kdc[23336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:52.146966 23688 init.cc:377] Logged in from keytab as kudu/127.27.54.66@KRBTEST.COM (short username kudu)
I20260807 06:53:52.148572 23688 webserver.cc:460] Webserver started at http://127.27.54.66:44935/ using document root <none> and password file <none>
I20260807 06:53:52.149398 23688 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:52.149561 23688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:52.149901 23688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:52.153103 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "523fe5124c2f4b3185a197ac62b37839"
format_stamp: "Formatted at 2026-08-07 06:53:52 on dist-test-slave-3kk6"
server_key: "60a46e881b6d84968e8b719017d1f4e2"
server_key_iv: "43fea354d0d0c35e5f9084b79d9b40d7"
server_key_version: "encryptionkey@0"
I20260807 06:53:52.154052 23688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "523fe5124c2f4b3185a197ac62b37839"
format_stamp: "Formatted at 2026-08-07 06:53:52 on dist-test-slave-3kk6"
server_key: "60a46e881b6d84968e8b719017d1f4e2"
server_key_iv: "43fea354d0d0c35e5f9084b79d9b40d7"
server_key_version: "encryptionkey@0"
I20260807 06:53:52.159811 23688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:53:52.164120 23707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:52.165512 23688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:53:52.165705 23688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "523fe5124c2f4b3185a197ac62b37839"
format_stamp: "Formatted at 2026-08-07 06:53:52 on dist-test-slave-3kk6"
server_key: "60a46e881b6d84968e8b719017d1f4e2"
server_key_iv: "43fea354d0d0c35e5f9084b79d9b40d7"
server_key_version: "encryptionkey@0"
I20260807 06:53:52.165937 23688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:52.196661 23688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:52.200986 23688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:52.201251 23688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:52.202226 23688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:52.203824 23688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:52.203903 23688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:52.204020 23688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:52.204098 23688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:52.239255 23688 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:35475
I20260807 06:53:52.239317 23872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:35475 every 8 connection(s)
I20260807 06:53:52.240634 23688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:52.245287 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 23688
I20260807 06:53:52.245532 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:53:52.246026 27865 external_mini_cluster.cc:1428] Setting key 4a8e44a23147aebca4a15bba3dfbdec8
Aug 07 06:53:52 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.66@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:52.262737 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.243536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:58797 (local address 127.27.54.126:34451)
0807 06:53:52.244004 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:52.244009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:52.245049 (+ 1040us) server_negotiation.cc:407] Connection header received
0807 06:53:52.247231 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:52.247237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:52.247340 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:52.247469 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:52.249114 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.249839 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.251513 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.251972 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.252111 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.255964 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:52.255969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:52.255972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:52.256005 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:52.258856 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:52.259733 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:52.259742 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:52.259747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:52.259941 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:52.260399 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:52.260407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:52.260413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:52.260890 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:52.261057 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:52.261894 (+ 837us) server_negotiation.cc:299] Negotiation successful
0807 06:53:52.262166 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260807 06:53:52.263859 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.244150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34451 (local address 127.27.54.66:58797)
0807 06:53:52.244870 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:52.244920 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:52.246912 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:52.247753 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:52.247766 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:52.248370 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:52.248942 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.248960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.249995 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.250001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.251349 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.251363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.252220 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.252228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.252463 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.253219 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:52.253242 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:52.255722 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:52.259014 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:52.259023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:52.259041 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:52.259572 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:52.260063 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:52.260066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:52.260069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:52.260265 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:52.261051 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:52.261058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:52.261498 (+ 440us) client_negotiation.cc:769] Sending connection context
0807 06:53:52.261734 (+ 236us) client_negotiation.cc:240] Negotiation successful
0807 06:53:52.262296 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":303,"threads_started":1}
I20260807 06:53:52.265595 23874 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34451
I20260807 06:53:52.265944 23874 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:52.266920 23874 heartbeater.cc:511] Master 127.27.54.126:34451 requested a full tablet report, sending...
I20260807 06:53:52.268687 23406 ts_manager.cc:194] Registered new tserver with Master: 523fe5124c2f4b3185a197ac62b37839 (127.27.54.66:35475)
I20260807 06:53:52.270015 23406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.66@KRBTEST.COM'} at 127.27.54.66:58797
WARNING: no policy specified for kudu/127.27.54.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
Entry for principal kudu/127.27.54.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab.
I20260807 06:53:52.325632 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34451
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:53:52.688117 23675 heartbeater.cc:503] Master 127.27.54.126:34451 was elected leader, sending a full tablet report...
W20260807 06:53:52.704950 23881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:52.705335 23881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:52.705518 23881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:52.712886 23881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:52.713065 23881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:52.713267 23881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:52.723667 23881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.67.keytab
--principal=kudu/127.27.54.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:34451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:52.725086 23881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:52.726436 23881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:52.740754 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:52.740792 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:52.740831 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:52.741508 23881 server_base.cc:1034] running on GCE node
I20260807 06:53:52.742086 23881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:52.743165 23881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:52.744374 23881 hybrid_clock.cc:648] HybridClock initialized: now 1786085632744308 us; error 60 us; skew 500 ppm
I20260807 06:53:52.744738 23881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:52 dist-test-slave-3kk6 krb5kdc[23336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:52.748136 23881 init.cc:377] Logged in from keytab as kudu/127.27.54.67@KRBTEST.COM (short username kudu)
I20260807 06:53:52.749734 23881 webserver.cc:460] Webserver started at http://127.27.54.67:44483/ using document root <none> and password file <none>
I20260807 06:53:52.750443 23881 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:52.750592 23881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:52.750978 23881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:52.753990 23881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "cf8be23cd2294e31931ff9fd456d65e1"
format_stamp: "Formatted at 2026-08-07 06:53:52 on dist-test-slave-3kk6"
server_key: "787366fb3e8a704c395725fd2fa54bbb"
server_key_iv: "f89efae554ce86ade59ad12197f7a3a5"
server_key_version: "encryptionkey@0"
I20260807 06:53:52.754902 23881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "cf8be23cd2294e31931ff9fd456d65e1"
format_stamp: "Formatted at 2026-08-07 06:53:52 on dist-test-slave-3kk6"
server_key: "787366fb3e8a704c395725fd2fa54bbb"
server_key_iv: "f89efae554ce86ade59ad12197f7a3a5"
server_key_version: "encryptionkey@0"
I20260807 06:53:52.760449 23881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:53:52.764495 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:52.765908 23881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:53:52.766110 23881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "cf8be23cd2294e31931ff9fd456d65e1"
format_stamp: "Formatted at 2026-08-07 06:53:52 on dist-test-slave-3kk6"
server_key: "787366fb3e8a704c395725fd2fa54bbb"
server_key_iv: "f89efae554ce86ade59ad12197f7a3a5"
server_key_version: "encryptionkey@0"
I20260807 06:53:52.766378 23881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:52.791849 23881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:52.796370 23881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:52.796715 23881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:52.797729 23881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:52.799216 23881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:52.799297 23881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:52.799387 23881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:52.799432 23881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:52.835745 23881 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:43443
I20260807 06:53:52.835817 24056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:43443 every 8 connection(s)
I20260807 06:53:52.837224 23881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:52.844501 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 23881
I20260807 06:53:52.844735 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:53:52.845209 27865 external_mini_cluster.cc:1428] Setting key 52594cd114a05a66137d0fd7058f6191
Aug 07 06:53:52 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085632, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.67@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:53:52.859658 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.840093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:43933 (local address 127.27.54.126:34451)
0807 06:53:52.840573 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:52.840578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:52.841736 (+ 1158us) server_negotiation.cc:407] Connection header received
0807 06:53:52.843832 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:52.843837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:52.844059 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:52.844203 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:52.846402 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.847467 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.849188 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.849751 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.849882 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.853683 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:52.853688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:52.853692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:52.853728 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:52.856357 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:52.857157 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:52.857161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:52.857164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:52.857300 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:52.857702 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:52.857705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:52.857707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:52.857989 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:52.858120 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:52.858959 (+ 839us) server_negotiation.cc:299] Negotiation successful
0807 06:53:52.859215 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260807 06:53:52.860962 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.840660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:34451 (local address 127.27.54.67:43933)
0807 06:53:52.841534 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:52.841591 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:52.843512 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:52.844488 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:52.844498 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:52.845336 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:52.846171 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.846205 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.847642 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.847646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.849048 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.849062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.850032 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.850037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.850252 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.851034 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:53:52.851059 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:52.853442 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:52.856533 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:52.856540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:52.856555 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:52.857041 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:52.857407 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:53:52.857410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:53:52.857412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:53:52.857603 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:53:52.858112 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:52.858118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:52.858486 (+ 368us) client_negotiation.cc:769] Sending connection context
0807 06:53:52.858850 (+ 364us) client_negotiation.cc:240] Negotiation successful
0807 06:53:52.859497 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":362,"threads_started":1}
I20260807 06:53:52.862545 24057 heartbeater.cc:347] Connected to a master server at 127.27.54.126:34451
I20260807 06:53:52.862944 24057 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:52.863746 24057 heartbeater.cc:511] Master 127.27.54.126:34451 requested a full tablet report, sending...
I20260807 06:53:52.865366 23406 ts_manager.cc:194] Registered new tserver with Master: cf8be23cd2294e31931ff9fd456d65e1 (127.27.54.67:43443)
I20260807 06:53:52.866246 23406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.54.67@KRBTEST.COM'} at 127.27.54.67:43933
I20260807 06:53:52.873943 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:53:52.892201 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.878610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.27.54.126:34451)
0807 06:53:52.878965 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:52.878976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:52.879432 (+ 456us) server_negotiation.cc:407] Connection header received
0807 06:53:52.879581 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:52.879585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:52.879701 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:52.879801 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:52.881412 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.882204 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.883621 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.884099 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.884214 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.886784 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:52.886789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:52.886792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:52.886826 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:52.889397 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:52.890099 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:52.890103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:52.890106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:52.890238 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:52.890598 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:52.890601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:52.890603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:52.891068 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:52.891272 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:52.891592 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 06:53:52.891812 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260807 06:53:52.896389 23406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 06:53:52.903481 23406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:53:52.954388 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.916863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:43443 (local address 127.0.0.1:53144)
0807 06:53:52.943741 (+ 26878us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:52.943773 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:52.944047 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:52.945582 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:52.945594 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:52.945642 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:52.946238 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.946257 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.948658 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.948664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.951434 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.951456 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.953252 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.953258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.953463 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.953531 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 06:53:52.953669 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 06:53:52.953882 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26658,"mutex_wait_us":57,"thread_start_us":29318,"threads_started":1}
I20260807 06:53:52.955379 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.917612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.27.54.67:43443)
0807 06:53:52.942823 (+ 25211us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:52.942837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:52.944240 (+ 1403us) server_negotiation.cc:407] Connection header received
0807 06:53:52.944395 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:52.944407 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:52.944792 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:52.945088 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:52.946765 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.948478 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.951637 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.953045 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.953203 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.953479 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:52.954853 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0807 06:53:52.955011 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24914,"thread_start_us":25257,"threads_started":1}
I20260807 06:53:52.955662 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.932455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.27.54.66:35475)
0807 06:53:52.946140 (+ 13685us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:52.946153 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:52.947166 (+ 1013us) server_negotiation.cc:407] Connection header received
0807 06:53:52.947518 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:52.947527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:52.947723 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:52.947892 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:52.948885 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.950152 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.952758 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.954228 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.954382 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.954569 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:52.955136 (+ 567us) server_negotiation.cc:299] Negotiation successful
0807 06:53:52.955307 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13525,"thread_start_us":13403,"threads_started":1}
I20260807 06:53:52.955160 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.916285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:32865 (local address 127.0.0.1:49958)
0807 06:53:52.930697 (+ 14412us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:52.930752 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:52.931000 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:52.947692 (+ 16692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:52.947696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:52.947727 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:52.948031 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.948039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.949953 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.949959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.952173 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.952185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.954201 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.954208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.954424 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.954456 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:53:52.954543 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 06:53:52.954723 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14075,"mutex_wait_us":121,"thread_start_us":14930,"threads_started":1}
I20260807 06:53:52.955925 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.916603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.27.54.65:32865)
0807 06:53:52.946818 (+ 30215us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:52.946826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:52.946858 (+ 32us) server_negotiation.cc:407] Connection header received
0807 06:53:52.946963 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:52.946971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:52.947228 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:52.947475 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:52.948219 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.949775 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.952333 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.953656 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.953808 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.954033 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:52.955301 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0807 06:53:52.955473 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30011,"thread_start_us":29864,"threads_started":1}
I20260807 06:53:52.955574 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:52.938385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:35475 (local address 127.0.0.1:52362)
0807 06:53:52.947022 (+ 8637us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:52.947045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:52.947287 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:52.948170 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:52.948174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:52.948199 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:52.948676 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.948696 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.950340 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.950345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.952573 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:52.952589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:52.954729 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:52.954735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:52.954923 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:52.954955 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:53:52.955056 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 06:53:52.955212 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8496,"mutex_wait_us":31,"thread_start_us":7890,"threads_started":1}
I20260807 06:53:52.960394 23780 tablet_service.cc:1467] Processing CreateTablet for tablet 75f45453ad374b2d9e3ca796e71b6118 (DEFAULT_TABLE table=test-table [id=98495c0f2d16402587c76d1abbd36a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:52.960340 23584 tablet_service.cc:1467] Processing CreateTablet for tablet 75f45453ad374b2d9e3ca796e71b6118 (DEFAULT_TABLE table=test-table [id=98495c0f2d16402587c76d1abbd36a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:52.960331 23972 tablet_service.cc:1467] Processing CreateTablet for tablet 75f45453ad374b2d9e3ca796e71b6118 (DEFAULT_TABLE table=test-table [id=98495c0f2d16402587c76d1abbd36a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:52.961848 23780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f45453ad374b2d9e3ca796e71b6118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:52.961932 23584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f45453ad374b2d9e3ca796e71b6118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:52.961956 23972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f45453ad374b2d9e3ca796e71b6118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:52.974198 24082 tablet_bootstrap.cc:492] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Bootstrap starting.
I20260807 06:53:52.977380 24085 tablet_bootstrap.cc:492] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Bootstrap starting.
I20260807 06:53:52.979467 24084 tablet_bootstrap.cc:492] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Bootstrap starting.
I20260807 06:53:52.980342 24082 tablet_bootstrap.cc:654] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:52.981069 24085 tablet_bootstrap.cc:654] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:52.981752 24082 log.cc:824] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:52.982323 24085 log.cc:824] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:52.982476 24084 tablet_bootstrap.cc:654] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:52.983745 24084 log.cc:824] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:52.985400 24085 tablet_bootstrap.cc:492] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: No bootstrap required, opened a new log
I20260807 06:53:52.985404 24082 tablet_bootstrap.cc:492] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: No bootstrap required, opened a new log
I20260807 06:53:52.985683 24082 ts_tablet_manager.cc:1397] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260807 06:53:52.985720 24085 ts_tablet_manager.cc:1397] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 06:53:52.987002 24084 tablet_bootstrap.cc:492] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: No bootstrap required, opened a new log
I20260807 06:53:52.987291 24084 ts_tablet_manager.cc:1397] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 06:53:52.993182 24082 raft_consensus.cc:348] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:52.993573 24082 raft_consensus.cc:374] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:52.993649 24082 raft_consensus.cc:729] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523fe5124c2f4b3185a197ac62b37839, State: Initialized, Role: FOLLOWER
I20260807 06:53:52.994228 24082 consensus_queue.cc:260] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:52.995452 23874 heartbeater.cc:503] Master 127.27.54.126:34451 was elected leader, sending a full tablet report...
I20260807 06:53:52.996261 24082 ts_tablet_manager.cc:1428] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Time spent starting tablet: real 0.010s user 0.009s sys 0.003s
I20260807 06:53:52.997675 24085 raft_consensus.cc:348] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:52.998230 24085 raft_consensus.cc:374] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:52.998400 24085 raft_consensus.cc:729] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef455e385a214c379cd235a567b23e7b, State: Initialized, Role: FOLLOWER
I20260807 06:53:52.998636 24084 raft_consensus.cc:348] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:52.999181 24084 raft_consensus.cc:374] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:52.999298 24084 raft_consensus.cc:729] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8be23cd2294e31931ff9fd456d65e1, State: Initialized, Role: FOLLOWER
I20260807 06:53:52.999253 24085 consensus_queue.cc:260] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:52.999918 24084 consensus_queue.cc:260] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:53.001776 24085 ts_tablet_manager.cc:1428] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260807 06:53:53.002955 24057 heartbeater.cc:503] Master 127.27.54.126:34451 was elected leader, sending a full tablet report...
I20260807 06:53:53.003530 24084 ts_tablet_manager.cc:1428] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260807 06:53:53.089022 24059 tablet.cc:2369] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:53:53.126981 24095 raft_consensus.cc:482] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:53.127344 24095 raft_consensus.cc:504] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:53.129213 24095 leader_election.cc:288] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 523fe5124c2f4b3185a197ac62b37839 (127.27.54.66:35475), cf8be23cd2294e31931ff9fd456d65e1 (127.27.54.67:43443)
I20260807 06:53:53.137463 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.129753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:45319 (local address 127.27.54.66:35475)
0807 06:53:53.129999 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:53.130006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:53.130698 (+ 692us) server_negotiation.cc:407] Connection header received
0807 06:53:53.131080 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:53.131086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:53.131164 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:53.131237 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:53.132133 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.133268 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.135487 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.136436 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.136564 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.136624 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:53.137046 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 06:53:53.137198 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 06:53:53.137621 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.129815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:54503 (local address 127.27.54.67:43443)
0807 06:53:53.130343 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:53.130382 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:53.131359 (+ 977us) server_negotiation.cc:407] Connection header received
0807 06:53:53.131600 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:53.131605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:53.131705 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:53.131783 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:53.132737 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.133790 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.135826 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.136660 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.136827 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.136898 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:53.137225 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 06:53:53.137371 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260807 06:53:53.137655 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.129768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:43443 (local address 127.27.54.65:54503)
0807 06:53:53.131217 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:53.131243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:53.131417 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:53.132018 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:53.132022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:53.132047 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:53.132523 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:53.132537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.133926 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.133930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:53.135670 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:53.135684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.136876 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.136880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:53.137053 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.137085 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 06:53:53.137185 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 06:53:53.137310 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"thread_start_us":918,"threads_started":1}
I20260807 06:53:53.138062 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.129467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:35475 (local address 127.27.54.65:45319)
0807 06:53:53.130538 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:53.130562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:53.130862 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:53.131466 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:53.131470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:53.131498 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:53.131943 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:53.131957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.133436 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.133441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:53.135295 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:53.135316 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.136663 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.136670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:53.136867 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.136903 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 06:53:53.137004 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 06:53:53.137098 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":54,"thread_start_us":2423,"threads_started":1}
I20260807 06:53:53.138821 23997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f45453ad374b2d9e3ca796e71b6118" candidate_uuid: "ef455e385a214c379cd235a567b23e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8be23cd2294e31931ff9fd456d65e1" is_pre_election: true
I20260807 06:53:53.139287 23811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f45453ad374b2d9e3ca796e71b6118" candidate_uuid: "ef455e385a214c379cd235a567b23e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "523fe5124c2f4b3185a197ac62b37839" is_pre_election: true
I20260807 06:53:53.139460 23997 raft_consensus.cc:2454] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef455e385a214c379cd235a567b23e7b in term 0.
I20260807 06:53:53.139835 23811 raft_consensus.cc:2454] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef455e385a214c379cd235a567b23e7b in term 0.
I20260807 06:53:53.140251 23526 leader_election.cc:302] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8be23cd2294e31931ff9fd456d65e1, ef455e385a214c379cd235a567b23e7b; no voters:
I20260807 06:53:53.140724 24095 raft_consensus.cc:2781] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:53:53.140882 24095 raft_consensus.cc:482] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:53:53.141021 24095 raft_consensus.cc:3037] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:53.143438 24095 raft_consensus.cc:504] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:53.144196 24095 leader_election.cc:288] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [CANDIDATE]: Term 1 election: Requested vote from peers 523fe5124c2f4b3185a197ac62b37839 (127.27.54.66:35475), cf8be23cd2294e31931ff9fd456d65e1 (127.27.54.67:43443)
I20260807 06:53:53.144632 23811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f45453ad374b2d9e3ca796e71b6118" candidate_uuid: "ef455e385a214c379cd235a567b23e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "523fe5124c2f4b3185a197ac62b37839"
I20260807 06:53:53.144766 23997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f45453ad374b2d9e3ca796e71b6118" candidate_uuid: "ef455e385a214c379cd235a567b23e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8be23cd2294e31931ff9fd456d65e1"
I20260807 06:53:53.144845 23811 raft_consensus.cc:3037] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:53.145009 23997 raft_consensus.cc:3037] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:53.147336 23811 raft_consensus.cc:2454] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef455e385a214c379cd235a567b23e7b in term 1.
I20260807 06:53:53.147344 23997 raft_consensus.cc:2454] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef455e385a214c379cd235a567b23e7b in term 1.
I20260807 06:53:53.147779 23526 leader_election.cc:302] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8be23cd2294e31931ff9fd456d65e1, ef455e385a214c379cd235a567b23e7b; no voters:
I20260807 06:53:53.148128 24095 raft_consensus.cc:2781] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:53.148661 24095 raft_consensus.cc:686] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 1 LEADER]: Becoming Leader. State: Replica: ef455e385a214c379cd235a567b23e7b, State: Running, Role: LEADER
I20260807 06:53:53.149139 24095 consensus_queue.cc:237] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } }
I20260807 06:53:53.153919 23405 catalog_manager.cc:5495] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ef455e385a214c379cd235a567b23e7b (127.27.54.65). New cstate: current_term: 1 leader_uuid: "ef455e385a214c379cd235a567b23e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef455e385a214c379cd235a567b23e7b" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 32865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 } health_report { overall_health: UNKNOWN } } }
W20260807 06:53:53.155694 23676 tablet.cc:2369] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:53:53.197907 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.192397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.27.54.65:32865)
0807 06:53:53.192643 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:53.192650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:53.192670 (+ 20us) server_negotiation.cc:407] Connection header received
0807 06:53:53.192775 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:53.192778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:53.192860 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:53.193019 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:53:53.193399 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.194156 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.195695 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.196128 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.196267 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.196793 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:53:53.197314 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:53:53.197496 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:53.197548 (+ 52us) server_negotiation.cc:299] Negotiation successful
0807 06:53:53.197633 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 06:53:53.206452 23811 raft_consensus.cc:1261] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 1 FOLLOWER]: Refusing update from remote peer ef455e385a214c379cd235a567b23e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:53.206450 23997 raft_consensus.cc:1261] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 1 FOLLOWER]: Refusing update from remote peer ef455e385a214c379cd235a567b23e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:53.207568 24099 consensus_queue.cc:1035] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf8be23cd2294e31931ff9fd456d65e1" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:53.207983 24095 consensus_queue.cc:1035] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "523fe5124c2f4b3185a197ac62b37839" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:53.220628 24109 mvcc.cc:205] Tried to move back new op lower bound from 7315806753602650112 to 7315806753382920192. Current Snapshot: MvccSnapshot[applied={T|T < 7315806753602650112}]
I20260807 06:53:53.221289 24105 mvcc.cc:205] Tried to move back new op lower bound from 7315806753602650112 to 7315806753382920192. Current Snapshot: MvccSnapshot[applied={T|T < 7315806753602650112}]
I20260807 06:53:53.224123 24106 mvcc.cc:205] Tried to move back new op lower bound from 7315806753602650112 to 7315806753382920192. Current Snapshot: MvccSnapshot[applied={T|T < 7315806753602650112}]
I20260807 06:53:53.236840 23406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:53.237227 23406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:53.241865 23406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 096d8e25f42944e3b61d2f02043ff1a3: Sending DeleteTablet for 3 replicas of tablet 75f45453ad374b2d9e3ca796e71b6118
I20260807 06:53:53.243296 23972 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f45453ad374b2d9e3ca796e71b6118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:53 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:53144
I20260807 06:53:53.244459 23780 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f45453ad374b2d9e3ca796e71b6118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:53 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:52362
I20260807 06:53:53.245644 23584 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f45453ad374b2d9e3ca796e71b6118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:53 UTC) from {username='kudu', principal='kudu/127.27.54.126@KRBTEST.COM'} at 127.0.0.1:49958
I20260807 06:53:53.249090 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 23501
I20260807 06:53:53.269646 24121 tablet_replica.cc:330] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: stopping tablet replica
I20260807 06:53:53.270330 24121 raft_consensus.cc:2229] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:53:53.271088 24121 raft_consensus.cc:2258] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:53.271899 24119 tablet_replica.cc:330] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: stopping tablet replica
I20260807 06:53:53.272310 24119 raft_consensus.cc:2229] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:53.272855 24119 raft_consensus.cc:2258] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:53.274624 24120 tablet_replica.cc:330] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: stopping tablet replica
I20260807 06:53:53.275122 24120 raft_consensus.cc:2229] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:53.275637 24120 raft_consensus.cc:2258] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:53.276005 24121 ts_tablet_manager.cc:1905] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:53.277755 24120 ts_tablet_manager.cc:1905] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:53.278383 24119 ts_tablet_manager.cc:1905] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:53.284478 24121 ts_tablet_manager.cc:1918] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:53.284708 24121 log.cc:1196] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/75f45453ad374b2d9e3ca796e71b6118
I20260807 06:53:53.284979 24120 ts_tablet_manager.cc:1918] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:53.285161 24120 log.cc:1196] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/75f45453ad374b2d9e3ca796e71b6118
I20260807 06:53:53.285394 24121 ts_tablet_manager.cc:1939] T 75f45453ad374b2d9e3ca796e71b6118 P ef455e385a214c379cd235a567b23e7b: Deleting consensus metadata
I20260807 06:53:53.285732 24120 ts_tablet_manager.cc:1939] T 75f45453ad374b2d9e3ca796e71b6118 P 523fe5124c2f4b3185a197ac62b37839: Deleting consensus metadata
I20260807 06:53:53.287604 23385 catalog_manager.cc:4841] TS 523fe5124c2f4b3185a197ac62b37839 (127.27.54.66:35475): tablet 75f45453ad374b2d9e3ca796e71b6118 (table test-table [id=98495c0f2d16402587c76d1abbd36a1c]) successfully deleted
I20260807 06:53:53.288007 23385 catalog_manager.cc:4841] TS ef455e385a214c379cd235a567b23e7b (127.27.54.65:32865): tablet 75f45453ad374b2d9e3ca796e71b6118 (table test-table [id=98495c0f2d16402587c76d1abbd36a1c]) successfully deleted
Aug 07 06:53:53 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.65@KRBTEST.COM
I20260807 06:53:53.292624 24119 ts_tablet_manager.cc:1918] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:53.292842 24119 log.cc:1196] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/75f45453ad374b2d9e3ca796e71b6118
I20260807 06:53:53.293413 24119 ts_tablet_manager.cc:1939] T 75f45453ad374b2d9e3ca796e71b6118 P cf8be23cd2294e31931ff9fd456d65e1: Deleting consensus metadata
I20260807 06:53:53.295588 23384 catalog_manager.cc:4841] TS cf8be23cd2294e31931ff9fd456d65e1 (127.27.54.67:43443): tablet 75f45453ad374b2d9e3ca796e71b6118 (table test-table [id=98495c0f2d16402587c76d1abbd36a1c]) successfully deleted
I20260807 06:53:53.299707 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.250770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.27.54.65:32865)
0807 06:53:53.250979 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:53.250986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:53.273933 (+ 22947us) server_negotiation.cc:407] Connection header received
0807 06:53:53.274184 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:53.274190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:53.274279 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:53.274386 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:53.278641 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.280025 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.284999 (+ 4974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.285750 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.285896 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.292306 (+ 6410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:53.292320 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:53.292335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:53.292385 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:53.296162 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:53.297053 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:53.297061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:53.297064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:53.297238 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:53.297862 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:53.297871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:53.297874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:53.298293 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:53.298637 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:53.299050 (+ 413us) server_negotiation.cc:299] Negotiation successful
0807 06:53:53.299323 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 06:53:53.300482 23669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:53.403486 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 23501
I20260807 06:53:53.433866 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 23688
Aug 07 06:53:53 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.66@KRBTEST.COM
I20260807 06:53:53.450785 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.434841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.27.54.66:35475)
0807 06:53:53.435048 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:53.435054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:53.435076 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:53:53.435189 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:53.435196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:53.435284 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:53.435400 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:53.436762 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.437530 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.439098 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.439502 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.439643 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.444261 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:53.444278 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:53.444293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:53.444330 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:53.447589 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:53.448386 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:53.448394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:53.448396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:53.448527 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:53.448982 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:53.448986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:53.448988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:53.449416 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:53.449736 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:53.450117 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 06:53:53.450372 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:53:53.451550 23868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:53.543428 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 23688
I20260807 06:53:53.565469 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 23881
Aug 07 06:53:53 dist-test-slave-3kk6 krb5kdc[23336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.67@KRBTEST.COM
I20260807 06:53:53.581692 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:53.566346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.27.54.67:43443)
0807 06:53:53.566589 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:53.566594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:53.566616 (+ 22us) server_negotiation.cc:407] Connection header received
0807 06:53:53.566903 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:53.566909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:53.566991 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:53.567117 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:53.568565 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.569318 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.570809 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:53.571298 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:53.571423 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:53.576132 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:53.576149 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:53.576178 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:53.576216 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:53.579025 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:53.579722 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:53.579729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:53.579732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:53.579866 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:53.580232 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:53.580236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:53.580238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:53.580596 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:53.580837 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:53.581111 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 06:53:53.581288 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 06:53:53.582360 24052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:53.673135 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 23881
I20260807 06:53:53.700712 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 23354
I20260807 06:53:53.701998 23444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:53.814815 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 23354
2026-08-07T06:53:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5146 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T06:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:53Z Disabled control of system clock
I20260807 06:53:53.924397 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39961
--webserver_interface=127.27.54.126
--webserver_port=0
--builtin_ntp_servers=127.27.54.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:53:54.291023 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:54.291353 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:54.291437 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:54.298849 24136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:54.298985 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:54.299036 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:54.299088 24136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:54.299134 24136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:54.308657 24136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:39961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:54.310011 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:54.311901 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:54.324220 24145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:54.324334 24144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:54.324270 24148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:54.325377 24136 server_base.cc:1034] running on GCE node
I20260807 06:53:54.326267 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:54.327775 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:54.328997 24136 hybrid_clock.cc:648] HybridClock initialized: now 1786085634328953 us; error 40 us; skew 500 ppm
I20260807 06:53:54.329401 24136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:54.331883 24136 webserver.cc:460] Webserver started at http://127.27.54.126:38977/ using document root <none> and password file <none>
I20260807 06:53:54.332612 24136 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:54.332741 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:54.333114 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:54.336341 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "6cf2bce2e93446e1a8a3307762c0ce6e"
format_stamp: "Formatted at 2026-08-07 06:53:54 on dist-test-slave-3kk6"
server_key: "e421d3d328dc747c743af4434c662494"
server_key_iv: "5b54214af18350ae7dbd6ff5ea18a7fe"
server_key_version: "encryptionkey@0"
I20260807 06:53:54.337239 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "6cf2bce2e93446e1a8a3307762c0ce6e"
format_stamp: "Formatted at 2026-08-07 06:53:54 on dist-test-slave-3kk6"
server_key: "e421d3d328dc747c743af4434c662494"
server_key_iv: "5b54214af18350ae7dbd6ff5ea18a7fe"
server_key_version: "encryptionkey@0"
I20260807 06:53:54.342911 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:53:54.346550 24155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:54.348007 24136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:53:54.348234 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "6cf2bce2e93446e1a8a3307762c0ce6e"
format_stamp: "Formatted at 2026-08-07 06:53:54 on dist-test-slave-3kk6"
server_key: "e421d3d328dc747c743af4434c662494"
server_key_iv: "5b54214af18350ae7dbd6ff5ea18a7fe"
server_key_version: "encryptionkey@0"
I20260807 06:53:54.348479 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:54.381362 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:54.382359 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:54.382772 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:54.403275 24136 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:39961
I20260807 06:53:54.403290 24226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:39961 every 8 connection(s)
I20260807 06:53:54.404803 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:54.408886 24227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:54.410255 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 24136
I20260807 06:53:54.410457 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:54.411026 27865 external_mini_cluster.cc:1428] Setting key ce0bf9f902f65e565e10de69664c0ebe
I20260807 06:53:54.424750 24227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e: Bootstrap starting.
I20260807 06:53:54.427438 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:54.413231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.27.54.126:39961)
0807 06:53:54.414525 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:54.414540 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:54.414595 (+ 55us) server_negotiation.cc:407] Connection header received
0807 06:53:54.416420 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:54.416461 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:54.417118 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:54.417638 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:54.418216 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:54.420209 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:54.421645 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:54.422796 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:54.422982 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:54.424228 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:54.424245 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:54.424264 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:54.425086 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:54.425205 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:54.425233 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:54.425348 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:54.425645 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 06:53:54.425993 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":455,"threads_started":1}
I20260807 06:53:54.428860 24227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:54.429998 24227 log.cc:824] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:54.433090 24227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e: No bootstrap required, opened a new log
I20260807 06:53:54.441077 24227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39961 } }
I20260807 06:53:54.441586 24227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:54.441713 24227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf2bce2e93446e1a8a3307762c0ce6e, State: Initialized, Role: FOLLOWER
I20260807 06:53:54.442474 24227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39961 } }
I20260807 06:53:54.442819 24227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:54.442994 24227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:54.443229 24227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:54.445631 24227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39961 } }
I20260807 06:53:54.446209 24227 leader_election.cc:302] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cf2bce2e93446e1a8a3307762c0ce6e; no voters:
I20260807 06:53:54.446928 24227 leader_election.cc:288] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:54.447055 24236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:54.447901 24236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [term 1 LEADER]: Becoming Leader. State: Replica: 6cf2bce2e93446e1a8a3307762c0ce6e, State: Running, Role: LEADER
I20260807 06:53:54.448405 24236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39961 } }
I20260807 06:53:54.449332 24227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:54.452114 24237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39961 } } }
I20260807 06:53:54.452456 24237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:54.451927 24239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf2bce2e93446e1a8a3307762c0ce6e. Latest consensus state: current_term: 1 leader_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cf2bce2e93446e1a8a3307762c0ce6e" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39961 } } }
I20260807 06:53:54.452625 24239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:54.454417 24245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:54.460486 24245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:54.471138 24245 catalog_manager.cc:1347] Generated new cluster ID: 24a26787efeb4b7f94beb096ec374cf7
I20260807 06:53:54.471333 24245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:54.521006 24245 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:54.522961 24245 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:54.530869 24245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e: Generated new TSK 0
I20260807 06:53:54.532043 24245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:53:54.544920 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.65:0
--local_ip_for_outbound_sockets=127.27.54.65
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39961
--builtin_ntp_servers=127.27.54.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 06:53:54.907912 24260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:54.908301 24260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:54.908460 24260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:54.916014 24260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:54.916196 24260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:54.916371 24260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.65
I20260807 06:53:54.926260 24260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.54.65
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.65
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:54.927769 24260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:54.929646 24260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:54.943735 24270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:54.943642 24272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:54.943701 24269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:54.944615 24260 server_base.cc:1034] running on GCE node
I20260807 06:53:54.945295 24260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:54.946287 24260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:54.947500 24260 hybrid_clock.cc:648] HybridClock initialized: now 1786085634947455 us; error 34 us; skew 500 ppm
I20260807 06:53:54.947793 24260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:54.950505 24260 webserver.cc:460] Webserver started at http://127.27.54.65:37161/ using document root <none> and password file <none>
I20260807 06:53:54.951270 24260 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:54.951402 24260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:54.951788 24260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:54.955392 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data/instance:
uuid: "51c8a7d1ec684687b2837b2d6f35a99d"
format_stamp: "Formatted at 2026-08-07 06:53:54 on dist-test-slave-3kk6"
server_key: "a7d3dd03b86e219fa911b9b4352df2f7"
server_key_iv: "70a2c8eb4d208f4e959511c5b4fdcba7"
server_key_version: "encryptionkey@0"
I20260807 06:53:54.956326 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance:
uuid: "51c8a7d1ec684687b2837b2d6f35a99d"
format_stamp: "Formatted at 2026-08-07 06:53:54 on dist-test-slave-3kk6"
server_key: "a7d3dd03b86e219fa911b9b4352df2f7"
server_key_iv: "70a2c8eb4d208f4e959511c5b4fdcba7"
server_key_version: "encryptionkey@0"
I20260807 06:53:54.962046 24260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 06:53:54.965927 24282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:54.967468 24260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260807 06:53:54.967670 24260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal
uuid: "51c8a7d1ec684687b2837b2d6f35a99d"
format_stamp: "Formatted at 2026-08-07 06:53:54 on dist-test-slave-3kk6"
server_key: "a7d3dd03b86e219fa911b9b4352df2f7"
server_key_iv: "70a2c8eb4d208f4e959511c5b4fdcba7"
server_key_version: "encryptionkey@0"
I20260807 06:53:54.967926 24260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:54.991464 24260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:54.992444 24260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:54.992806 24260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:54.993958 24260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:54.995645 24260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:54.995743 24260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:54.995908 24260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:54.996011 24260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:55.031812 24260 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.65:33123
I20260807 06:53:55.031893 24444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.65:33123 every 8 connection(s)
I20260807 06:53:55.033208 24260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/data/info.pb
I20260807 06:53:55.033720 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 24260
I20260807 06:53:55.033911 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal/instance
I20260807 06:53:55.034538 27865 external_mini_cluster.cc:1428] Setting key 8df9f72992440bb5833b939e1f07d8dd
I20260807 06:53:55.037674 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.66:0
--local_ip_for_outbound_sockets=127.27.54.66
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39961
--builtin_ntp_servers=127.27.54.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:53:55.047499 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:55.036250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.65:39109 (local address 127.27.54.126:39961)
0807 06:53:55.036753 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:55.036757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:55.038180 (+ 1423us) server_negotiation.cc:407] Connection header received
0807 06:53:55.040073 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:55.040083 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:55.040177 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:55.040307 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:55.041370 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.042538 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.044160 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.044867 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.045030 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:55.046424 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:55.046433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:55.046440 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:55.046467 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:55.046522 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:55.046542 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:55.046643 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:55.046935 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 06:53:55.047051 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":285,"threads_started":1}
I20260807 06:53:55.048270 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:55.036929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39961 (local address 127.27.54.65:39109)
0807 06:53:55.037998 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:55.038051 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:55.039793 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:55.040501 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:55.040509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:55.040597 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:55.041189 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:55.041206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.042751 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.042754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:55.044004 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:55.044018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.045105 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.045109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:55.045315 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:55.046089 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:55.046091 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:55.046129 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:55.046211 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:55.046644 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:55.046695 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:55.046705 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 06:53:55.046880 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 06:53:55.047177 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":362,"threads_started":1}
I20260807 06:53:55.050014 24445 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39961
I20260807 06:53:55.050335 24445 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:55.051195 24445 heartbeater.cc:511] Master 127.27.54.126:39961 requested a full tablet report, sending...
I20260807 06:53:55.053619 24182 ts_manager.cc:194] Registered new tserver with Master: 51c8a7d1ec684687b2837b2d6f35a99d (127.27.54.65:33123)
I20260807 06:53:55.056021 24182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.65:39109
W20260807 06:53:55.395462 24452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:55.395778 24452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:55.395869 24452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:55.403443 24452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:55.403563 24452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:55.403734 24452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.66
I20260807 06:53:55.413223 24452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.54.66
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.66
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:55.414570 24452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:55.416527 24452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:55.430480 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:55.430510 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:55.430807 24462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:55.431392 24452 server_base.cc:1034] running on GCE node
I20260807 06:53:55.432006 24452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:55.432967 24452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:55.434176 24452 hybrid_clock.cc:648] HybridClock initialized: now 1786085635434113 us; error 59 us; skew 500 ppm
I20260807 06:53:55.434533 24452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:55.436924 24452 webserver.cc:460] Webserver started at http://127.27.54.66:34157/ using document root <none> and password file <none>
I20260807 06:53:55.437669 24452 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:55.437793 24452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:55.438148 24452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:55.441407 24452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data/instance:
uuid: "7415d0591c2b41f7b0897aaf9b04725d"
format_stamp: "Formatted at 2026-08-07 06:53:55 on dist-test-slave-3kk6"
server_key: "aad22d65e1fbbec29d57e1dc02544cd2"
server_key_iv: "9c6e59ae10aedc73cc83b77223265a0a"
server_key_version: "encryptionkey@0"
I20260807 06:53:55.442265 24452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance:
uuid: "7415d0591c2b41f7b0897aaf9b04725d"
format_stamp: "Formatted at 2026-08-07 06:53:55 on dist-test-slave-3kk6"
server_key: "aad22d65e1fbbec29d57e1dc02544cd2"
server_key_iv: "9c6e59ae10aedc73cc83b77223265a0a"
server_key_version: "encryptionkey@0"
I20260807 06:53:55.447697 24452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 06:53:55.451404 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:55.452905 24452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:53:55.453048 24452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal
uuid: "7415d0591c2b41f7b0897aaf9b04725d"
format_stamp: "Formatted at 2026-08-07 06:53:55 on dist-test-slave-3kk6"
server_key: "aad22d65e1fbbec29d57e1dc02544cd2"
server_key_iv: "9c6e59ae10aedc73cc83b77223265a0a"
server_key_version: "encryptionkey@0"
I20260807 06:53:55.453244 24452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:55.479063 24452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:55.479941 24452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:55.480233 24452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:55.481256 24452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:55.482887 24452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:55.482987 24452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:55.483070 24452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:55.483131 24452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:55.518608 24452 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.66:36481
I20260807 06:53:55.518748 24615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.66:36481 every 8 connection(s)
I20260807 06:53:55.520076 24452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/data/info.pb
I20260807 06:53:55.527207 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 24452
I20260807 06:53:55.527406 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal/instance
I20260807 06:53:55.527907 27865 external_mini_cluster.cc:1428] Setting key 80f8074fcbd194e8b77dcbf6287e66f8
I20260807 06:53:55.531296 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.54.67:0
--local_ip_for_outbound_sockets=127.27.54.67
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39961
--builtin_ntp_servers=127.27.54.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 06:53:55.533803 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:55.523036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:33623 (local address 127.27.54.126:39961)
0807 06:53:55.523272 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:55.523278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:55.524622 (+ 1344us) server_negotiation.cc:407] Connection header received
0807 06:53:55.526455 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:55.526459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:55.526532 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:55.526627 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:55.528065 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.529076 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.530726 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.531182 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.531394 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:55.532919 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:55.532925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:55.532932 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:55.532956 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:55.532989 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:55.533001 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:55.533105 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:55.533366 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 06:53:55.533481 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 06:53:55.535074 24620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:55.523575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39961 (local address 127.27.54.66:33623)
0807 06:53:55.524456 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:55.524493 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:55.526173 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:55.526953 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:55.526966 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:55.527097 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:55.527859 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:55.527878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.529241 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.529253 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:55.530538 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:55.530551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:55.531489 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:55.531496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:55.531790 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:55.532569 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:55.532571 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:55.532614 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:55.532701 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:55.533107 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:55.533121 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:55.533131 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 06:53:55.533356 (+ 225us) client_negotiation.cc:240] Negotiation successful
0807 06:53:55.533799 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":270,"threads_started":1}
I20260807 06:53:55.536764 24617 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39961
I20260807 06:53:55.537127 24617 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:55.538089 24617 heartbeater.cc:511] Master 127.27.54.126:39961 requested a full tablet report, sending...
I20260807 06:53:55.539908 24182 ts_manager.cc:194] Registered new tserver with Master: 7415d0591c2b41f7b0897aaf9b04725d (127.27.54.66:36481)
I20260807 06:53:55.541190 24182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.66:33623
W20260807 06:53:55.896648 24621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:55.897014 24621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:55.897179 24621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:55.904772 24621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:55.904924 24621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:55.905102 24621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.54.67
I20260807 06:53:55.914543 24621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.54.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.54.67
--webserver_port=0
--tserver_master_addrs=127.27.54.126:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.54.67
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:55.915903 24621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:55.917730 24621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:55.931175 24628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:55.931270 24627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:55.931200 24631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:55.932315 24621 server_base.cc:1034] running on GCE node
I20260807 06:53:55.933033 24621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:55.934082 24621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:55.935323 24621 hybrid_clock.cc:648] HybridClock initialized: now 1786085635935254 us; error 61 us; skew 500 ppm
I20260807 06:53:55.935675 24621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 06:53:55.941370 24621 webserver.cc:460] Webserver started at http://127.27.54.67:43031/ using document root <none> and password file <none>
I20260807 06:53:55.942145 24621 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:55.942291 24621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:55.942728 24621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:55.945785 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data/instance:
uuid: "f88fe79f530043a18615181f52662f05"
format_stamp: "Formatted at 2026-08-07 06:53:55 on dist-test-slave-3kk6"
server_key: "08d3dbacfc17b2eec9f373ff671981d3"
server_key_iv: "b91f60e894475a4a6572076af8127855"
server_key_version: "encryptionkey@0"
I20260807 06:53:55.946631 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance:
uuid: "f88fe79f530043a18615181f52662f05"
format_stamp: "Formatted at 2026-08-07 06:53:55 on dist-test-slave-3kk6"
server_key: "08d3dbacfc17b2eec9f373ff671981d3"
server_key_iv: "b91f60e894475a4a6572076af8127855"
server_key_version: "encryptionkey@0"
I20260807 06:53:55.952358 24621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 06:53:55.956135 24638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:55.957545 24621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 06:53:55.957731 24621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal
uuid: "f88fe79f530043a18615181f52662f05"
format_stamp: "Formatted at 2026-08-07 06:53:55 on dist-test-slave-3kk6"
server_key: "08d3dbacfc17b2eec9f373ff671981d3"
server_key_iv: "b91f60e894475a4a6572076af8127855"
server_key_version: "encryptionkey@0"
I20260807 06:53:55.957933 24621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:55.973387 24621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:55.974193 24621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:55.974462 24621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:55.975528 24621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 06:53:55.977043 24621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 06:53:55.977135 24621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:55.977227 24621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 06:53:55.977288 24621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:56.012316 24621 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.67:38769
I20260807 06:53:56.012415 24780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.67:38769 every 8 connection(s)
I20260807 06:53:56.013764 24621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/data/info.pb
I20260807 06:53:56.019925 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 24621
I20260807 06:53:56.020145 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal/instance
I20260807 06:53:56.020653 27865 external_mini_cluster.cc:1428] Setting key 22f9f186d63d98c4e3d959d54d33abf9
I20260807 06:53:56.027207 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.016498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.67:35351 (local address 127.27.54.126:39961)
0807 06:53:56.016690 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.016694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.018045 (+ 1351us) server_negotiation.cc:407] Connection header received
0807 06:53:56.020033 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.020040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.020142 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.020266 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:56.021383 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.022501 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.024579 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.024977 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.025073 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.026513 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:56.026518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:56.026521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:56.026538 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:56.026567 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:56.026578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:56.026645 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.026888 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.026967 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 06:53:56.028429 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.016973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:39961 (local address 127.27.54.67:35351)
0807 06:53:56.017863 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:56.017912 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:56.019688 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:56.020498 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:56.020507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:56.020593 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:53:56.021225 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.021243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.022714 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.022722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.024428 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.024443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.025219 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.025225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.025437 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.026217 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 06:53:56.026220 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:53:56.026261 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 06:53:56.026340 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:53:56.026643 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:53:56.026698 (+ 55us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:53:56.026710 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 06:53:56.026865 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:53:56.027173 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260807 06:53:56.029922 24781 heartbeater.cc:347] Connected to a master server at 127.27.54.126:39961
I20260807 06:53:56.030244 24781 heartbeater.cc:464] Registering TS with master...
I20260807 06:53:56.031113 24781 heartbeater.cc:511] Master 127.27.54.126:39961 requested a full tablet report, sending...
I20260807 06:53:56.032653 24182 ts_manager.cc:194] Registered new tserver with Master: f88fe79f530043a18615181f52662f05 (127.27.54.67:38769)
I20260807 06:53:56.033483 24182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.54.67:35351
I20260807 06:53:56.037940 27865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 06:53:56.048066 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.042292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.27.54.126:39961)
0807 06:53:56.042508 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.042517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.043062 (+ 545us) server_negotiation.cc:407] Connection header received
0807 06:53:56.043197 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.043200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.043261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.043347 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:56.043757 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.044450 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.045770 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.046166 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.046257 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.047385 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:56.047389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:56.047392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:56.047409 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:56.047438 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:56.047449 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:56.047519 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.047647 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.047718 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:53:56.052148 24182 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 06:53:56.059377 24445 heartbeater.cc:503] Master 127.27.54.126:39961 was elected leader, sending a full tablet report...
W20260807 06:53:56.059496 24182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 06:53:56.084299 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.073651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33123 (local address 127.0.0.1:48378)
0807 06:53:56.074225 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:56.074243 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:56.074364 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:56.075297 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:56.075305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:56.075349 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:56.076448 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.076463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.078351 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.078356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.080643 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.080653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.083046 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.083051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.083353 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.083477 (+ 124us) client_negotiation.cc:769] Sending connection context
0807 06:53:56.083611 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 06:53:56.083842 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":691,"spinlock_wait_cycles":4992,"thread_start_us":287,"threads_started":1}
I20260807 06:53:56.084316 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.072308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.27.54.66:36481)
0807 06:53:56.073041 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.073051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.074371 (+ 1320us) server_negotiation.cc:407] Connection header received
0807 06:53:56.074490 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.074500 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.074782 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.074988 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:56.076133 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.077772 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.080915 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.082439 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.082580 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.082860 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.083759 (+ 899us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.083946 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":404,"threads_started":1}
I20260807 06:53:56.085412 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.073873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.27.54.65:33123)
0807 06:53:56.074450 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.074458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.074482 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:53:56.074563 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.074569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.074789 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.074988 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:56.076656 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.078165 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.080907 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.082439 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.082580 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.082894 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.084828 (+ 1934us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.085029 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260807 06:53:56.085011 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.072185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.66:36481 (local address 127.0.0.1:52376)
0807 06:53:56.073275 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:56.073329 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:56.073522 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:56.075297 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:56.075305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:56.075353 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:56.075872 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.075887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.077951 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.077956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.080635 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.080653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.083034 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.083041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.083350 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.083477 (+ 127us) client_negotiation.cc:769] Sending connection context
0807 06:53:56.083611 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 06:53:56.084649 (+ 1038us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":989,"spinlock_wait_cycles":14080,"thread_start_us":617,"threads_started":1}
I20260807 06:53:56.089534 24351 tablet_service.cc:1467] Processing CreateTablet for tablet c547d30ad230431c87a8265a056d73b5 (DEFAULT_TABLE table=test-table [id=ddda23175812482da5049b249bfd98fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:56.089540 24527 tablet_service.cc:1467] Processing CreateTablet for tablet c547d30ad230431c87a8265a056d73b5 (DEFAULT_TABLE table=test-table [id=ddda23175812482da5049b249bfd98fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:56.091041 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c547d30ad230431c87a8265a056d73b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:56.091065 24351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c547d30ad230431c87a8265a056d73b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:56.093912 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.074298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:38769 (local address 127.0.0.1:40562)
0807 06:53:56.084104 (+ 9806us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:56.084123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:56.084249 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:56.086455 (+ 2206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:56.086459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:56.086490 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:56.086892 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.086904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.089042 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.089048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.091178 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.091194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.093077 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.093087 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.093277 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.093315 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 06:53:56.093417 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 06:53:56.093555 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9661,"thread_start_us":10045,"threads_started":1}
I20260807 06:53:56.094045 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.074967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.27.54.67:38769)
0807 06:53:56.085768 (+ 10801us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.085776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.085806 (+ 30us) server_negotiation.cc:407] Connection header received
0807 06:53:56.085951 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.085960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.086236 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.086437 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:56.087093 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.088665 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.091401 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.092849 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.093003 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.093292 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.093482 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.093650 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10475,"thread_start_us":10282,"threads_started":1}
I20260807 06:53:56.098680 24699 tablet_service.cc:1467] Processing CreateTablet for tablet c547d30ad230431c87a8265a056d73b5 (DEFAULT_TABLE table=test-table [id=ddda23175812482da5049b249bfd98fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 06:53:56.100528 24699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c547d30ad230431c87a8265a056d73b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:53:56.104940 24803 tablet_bootstrap.cc:492] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Bootstrap starting.
I20260807 06:53:56.108891 24803 tablet_bootstrap.cc:654] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:56.110261 24803 log.cc:824] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:56.110910 24804 tablet_bootstrap.cc:492] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Bootstrap starting.
I20260807 06:53:56.114212 24804 tablet_bootstrap.cc:654] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:56.114328 24803 tablet_bootstrap.cc:492] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: No bootstrap required, opened a new log
I20260807 06:53:56.114701 24803 ts_tablet_manager.cc:1397] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260807 06:53:56.115613 24807 tablet_bootstrap.cc:492] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Bootstrap starting.
I20260807 06:53:56.116842 24804 log.cc:824] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:56.119298 24807 tablet_bootstrap.cc:654] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:56.120333 24804 tablet_bootstrap.cc:492] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: No bootstrap required, opened a new log
I20260807 06:53:56.120636 24804 ts_tablet_manager.cc:1397] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260807 06:53:56.120604 24807 log.cc:824] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:56.123834 24807 tablet_bootstrap.cc:492] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: No bootstrap required, opened a new log
I20260807 06:53:56.124141 24807 ts_tablet_manager.cc:1397] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 06:53:56.124315 24803 raft_consensus.cc:348] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.124877 24803 raft_consensus.cc:374] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:56.125010 24803 raft_consensus.cc:729] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7415d0591c2b41f7b0897aaf9b04725d, State: Initialized, Role: FOLLOWER
I20260807 06:53:56.125720 24803 consensus_queue.cc:260] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.126994 24617 heartbeater.cc:503] Master 127.27.54.126:39961 was elected leader, sending a full tablet report...
I20260807 06:53:56.127719 24803 ts_tablet_manager.cc:1428] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260807 06:53:56.130927 24804 raft_consensus.cc:348] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.131361 24804 raft_consensus.cc:374] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:56.131491 24804 raft_consensus.cc:729] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c8a7d1ec684687b2837b2d6f35a99d, State: Initialized, Role: FOLLOWER
I20260807 06:53:56.132220 24804 consensus_queue.cc:260] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.134295 24804 ts_tablet_manager.cc:1428] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 06:53:56.135426 24807 raft_consensus.cc:348] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.135815 24807 raft_consensus.cc:374] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:56.135927 24807 raft_consensus.cc:729] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88fe79f530043a18615181f52662f05, State: Initialized, Role: FOLLOWER
I20260807 06:53:56.136579 24807 consensus_queue.cc:260] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.137814 24781 heartbeater.cc:503] Master 127.27.54.126:39961 was elected leader, sending a full tablet report...
I20260807 06:53:56.138456 24807 ts_tablet_manager.cc:1428] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260807 06:53:56.147128 24812 raft_consensus.cc:482] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 06:53:56.147460 24812 raft_consensus.cc:504] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.149089 24812 leader_election.cc:288] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51c8a7d1ec684687b2837b2d6f35a99d (127.27.54.65:33123), f88fe79f530043a18615181f52662f05 (127.27.54.67:38769)
I20260807 06:53:56.159031 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.150515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:35657 (local address 127.27.54.67:38769)
0807 06:53:56.150855 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.150861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.151074 (+ 213us) server_negotiation.cc:407] Connection header received
0807 06:53:56.151220 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.151224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.151283 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.151362 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:56.152220 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.153330 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.156835 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.157832 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.157970 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.158020 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.158561 (+ 541us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.158754 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260807 06:53:56.159030 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.149685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.54.66:44493 (local address 127.27.54.65:33123)
0807 06:53:56.149960 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.149966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.150237 (+ 271us) server_negotiation.cc:407] Connection header received
0807 06:53:56.150574 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.150582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.150952 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.151040 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 06:53:56.152126 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.153330 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.156797 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.157853 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.158002 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.158069 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.158646 (+ 577us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.158800 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 06:53:56.159039 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.150280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.67:38769 (local address 127.27.54.66:35657)
0807 06:53:56.150963 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:56.150987 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:56.151105 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:56.151410 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:56.151419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:56.151454 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:56.152050 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.152064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.153497 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.153503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.156508 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.156525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.158019 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.158026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.158281 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.158315 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 06:53:56.158428 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 06:53:56.158576 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":246,"thread_start_us":390,"threads_started":1}
I20260807 06:53:56.159066 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.149450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.65:33123 (local address 127.27.54.66:44493)
0807 06:53:56.150112 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0807 06:53:56.150129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 06:53:56.150335 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:53:56.151272 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:53:56.151275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:53:56.151295 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 06:53:56.151890 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.151906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.153518 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.153538 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.156508 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:53:56.156525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.158269 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.158276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:53:56.158455 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.158483 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 06:53:56.158561 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 06:53:56.158705 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":236,"thread_start_us":354,"threads_started":1}
I20260807 06:53:56.160373 24722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c547d30ad230431c87a8265a056d73b5" candidate_uuid: "7415d0591c2b41f7b0897aaf9b04725d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88fe79f530043a18615181f52662f05" is_pre_election: true
I20260807 06:53:56.161033 24722 raft_consensus.cc:2454] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7415d0591c2b41f7b0897aaf9b04725d in term 0.
I20260807 06:53:56.161281 24381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c547d30ad230431c87a8265a056d73b5" candidate_uuid: "7415d0591c2b41f7b0897aaf9b04725d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" is_pre_election: true
I20260807 06:53:56.161823 24473 leader_election.cc:302] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7415d0591c2b41f7b0897aaf9b04725d, f88fe79f530043a18615181f52662f05; no voters:
I20260807 06:53:56.161868 24381 raft_consensus.cc:2454] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7415d0591c2b41f7b0897aaf9b04725d in term 0.
I20260807 06:53:56.162303 24812 raft_consensus.cc:2781] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 06:53:56.162539 24812 raft_consensus.cc:482] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 06:53:56.162721 24812 raft_consensus.cc:3037] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:56.165076 24812 raft_consensus.cc:504] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.165683 24812 leader_election.cc:288] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [CANDIDATE]: Term 1 election: Requested vote from peers 51c8a7d1ec684687b2837b2d6f35a99d (127.27.54.65:33123), f88fe79f530043a18615181f52662f05 (127.27.54.67:38769)
I20260807 06:53:56.166204 24381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c547d30ad230431c87a8265a056d73b5" candidate_uuid: "7415d0591c2b41f7b0897aaf9b04725d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c8a7d1ec684687b2837b2d6f35a99d"
I20260807 06:53:56.166257 24722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c547d30ad230431c87a8265a056d73b5" candidate_uuid: "7415d0591c2b41f7b0897aaf9b04725d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88fe79f530043a18615181f52662f05"
I20260807 06:53:56.166414 24381 raft_consensus.cc:3037] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:56.166496 24722 raft_consensus.cc:3037] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:56.168810 24381 raft_consensus.cc:2454] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7415d0591c2b41f7b0897aaf9b04725d in term 1.
I20260807 06:53:56.168994 24722 raft_consensus.cc:2454] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7415d0591c2b41f7b0897aaf9b04725d in term 1.
I20260807 06:53:56.169287 24473 leader_election.cc:302] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c8a7d1ec684687b2837b2d6f35a99d, 7415d0591c2b41f7b0897aaf9b04725d; no voters:
I20260807 06:53:56.169633 24812 raft_consensus.cc:2781] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:56.170199 24812 raft_consensus.cc:686] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 1 LEADER]: Becoming Leader. State: Replica: 7415d0591c2b41f7b0897aaf9b04725d, State: Running, Role: LEADER
I20260807 06:53:56.170760 24812 consensus_queue.cc:237] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } }
I20260807 06:53:56.176170 24182 catalog_manager.cc:5495] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7415d0591c2b41f7b0897aaf9b04725d (127.27.54.66). New cstate: current_term: 1 leader_uuid: "7415d0591c2b41f7b0897aaf9b04725d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7415d0591c2b41f7b0897aaf9b04725d" member_type: VOTER last_known_addr { host: "127.27.54.66" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 } health_report { overall_health: UNKNOWN } } }
I20260807 06:53:56.213678 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.208131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.27.54.66:36481)
0807 06:53:56.208395 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.208399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.208415 (+ 16us) server_negotiation.cc:407] Connection header received
0807 06:53:56.208508 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.208512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.208603 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.208773 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:53:56.209183 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.209909 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.211419 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.211877 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.212022 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.212537 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:53:56.213039 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:53:56.213208 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.213318 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.213405 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 06:53:56.222281 24722 raft_consensus.cc:1261] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 1 FOLLOWER]: Refusing update from remote peer 7415d0591c2b41f7b0897aaf9b04725d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:56.222292 24381 raft_consensus.cc:1261] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 1 FOLLOWER]: Refusing update from remote peer 7415d0591c2b41f7b0897aaf9b04725d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:53:56.223361 24821 consensus_queue.cc:1035] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f88fe79f530043a18615181f52662f05" member_type: VOTER last_known_addr { host: "127.27.54.67" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:56.223702 24812 consensus_queue.cc:1035] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c8a7d1ec684687b2837b2d6f35a99d" member_type: VOTER last_known_addr { host: "127.27.54.65" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:53:56.234899 24828 mvcc.cc:205] Tried to move back new op lower bound from 7315806765956218880 to 7315806765760241664. Current Snapshot: MvccSnapshot[applied={T|T < 7315806765956218880}]
I20260807 06:53:56.234953 24824 mvcc.cc:205] Tried to move back new op lower bound from 7315806765956218880 to 7315806765760241664. Current Snapshot: MvccSnapshot[applied={T|T < 7315806765956218880}]
I20260807 06:53:56.250591 24179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:56.251008 24179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 06:53:56.254930 24179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cf2bce2e93446e1a8a3307762c0ce6e: Sending DeleteTablet for 3 replicas of tablet c547d30ad230431c87a8265a056d73b5
I20260807 06:53:56.256204 24351 tablet_service.cc:1514] Processing DeleteTablet for tablet c547d30ad230431c87a8265a056d73b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:56 UTC) from {username='slave'} at 127.0.0.1:48378
I20260807 06:53:56.256319 24527 tablet_service.cc:1514] Processing DeleteTablet for tablet c547d30ad230431c87a8265a056d73b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:56 UTC) from {username='slave'} at 127.0.0.1:52376
I20260807 06:53:56.256701 24699 tablet_service.cc:1514] Processing DeleteTablet for tablet c547d30ad230431c87a8265a056d73b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 06:53:56 UTC) from {username='slave'} at 127.0.0.1:40562
I20260807 06:53:56.258126 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 24260
I20260807 06:53:56.269146 24841 tablet_replica.cc:330] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: stopping tablet replica
I20260807 06:53:56.269827 24841 raft_consensus.cc:2229] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:56.270563 24841 raft_consensus.cc:2258] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:56.273087 24842 tablet_replica.cc:330] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: stopping tablet replica
I20260807 06:53:56.273732 24844 tablet_replica.cc:330] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: stopping tablet replica
I20260807 06:53:56.274176 24842 raft_consensus.cc:2229] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 1 LEADER]: Raft consensus shutting down.
I20260807 06:53:56.274395 24844 raft_consensus.cc:2229] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 06:53:56.275074 24842 raft_consensus.cc:2258] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:56.275902 24844 raft_consensus.cc:2258] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 06:53:56.277024 24841 ts_tablet_manager.cc:1905] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:56.277733 24842 ts_tablet_manager.cc:1905] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:56.277894 24844 ts_tablet_manager.cc:1905] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 06:53:56.285172 24844 ts_tablet_manager.cc:1918] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:56.285192 24841 ts_tablet_manager.cc:1918] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:56.285380 24841 log.cc:1196] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-0/wal/wals/c547d30ad230431c87a8265a056d73b5
I20260807 06:53:56.285467 24844 log.cc:1196] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-2/wal/wals/c547d30ad230431c87a8265a056d73b5
I20260807 06:53:56.286060 24841 ts_tablet_manager.cc:1939] T c547d30ad230431c87a8265a056d73b5 P 51c8a7d1ec684687b2837b2d6f35a99d: Deleting consensus metadata
I20260807 06:53:56.286158 24844 ts_tablet_manager.cc:1939] T c547d30ad230431c87a8265a056d73b5 P f88fe79f530043a18615181f52662f05: Deleting consensus metadata
I20260807 06:53:56.287745 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.259616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.27.54.65:33123)
0807 06:53:56.259801 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.259810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.274525 (+ 14715us) server_negotiation.cc:407] Connection header received
0807 06:53:56.274925 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.274933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.275045 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.275167 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:56.278105 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.279379 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.282789 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.283521 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.283636 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.285898 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:56.285910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:56.285928 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:56.286752 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:56.286826 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:56.286858 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:56.286990 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.287232 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.287354 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:53:56.288239 24157 catalog_manager.cc:4841] TS f88fe79f530043a18615181f52662f05 (127.27.54.67:38769): tablet c547d30ad230431c87a8265a056d73b5 (table test-table [id=ddda23175812482da5049b249bfd98fd]) successfully deleted
I20260807 06:53:56.288609 24157 catalog_manager.cc:4841] TS 51c8a7d1ec684687b2837b2d6f35a99d (127.27.54.65:33123): tablet c547d30ad230431c87a8265a056d73b5 (table test-table [id=ddda23175812482da5049b249bfd98fd]) successfully deleted
I20260807 06:53:56.289218 24439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:56.293488 24842 ts_tablet_manager.cc:1918] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 06:53:56.293884 24842 log.cc:1196] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Deleting WAL directory at /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786085287136147-27865-0/minicluster-data/ts-1/wal/wals/c547d30ad230431c87a8265a056d73b5
I20260807 06:53:56.294682 24842 ts_tablet_manager.cc:1939] T c547d30ad230431c87a8265a056d73b5 P 7415d0591c2b41f7b0897aaf9b04725d: Deleting consensus metadata
I20260807 06:53:56.296911 24157 catalog_manager.cc:4841] TS 7415d0591c2b41f7b0897aaf9b04725d (127.27.54.66:36481): tablet c547d30ad230431c87a8265a056d73b5 (table test-table [id=ddda23175812482da5049b249bfd98fd]) successfully deleted
I20260807 06:53:56.416249 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 24260
I20260807 06:53:56.442116 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 24452
I20260807 06:53:56.450352 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.443124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.27.54.66:36481)
0807 06:53:56.443321 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.443325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.443363 (+ 38us) server_negotiation.cc:407] Connection header received
0807 06:53:56.443584 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.443588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.443703 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.443796 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:56.444299 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.445096 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.446990 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.447408 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.447507 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.448764 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:56.448773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:56.448790 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:56.449542 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:56.449595 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:56.449616 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:56.449708 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.449879 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.449961 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:53:56.451165 24610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:56.568411 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 24452
I20260807 06:53:56.597537 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 24621
I20260807 06:53:56.605039 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:56.598460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.27.54.67:38769)
0807 06:53:56.598756 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:56.598761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:56.598782 (+ 21us) server_negotiation.cc:407] Connection header received
0807 06:53:56.598961 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:56.598965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:56.599071 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:56.599165 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:56.599627 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.600444 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.601881 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:56.602318 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:56.602432 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:56.603608 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:56.603618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:56.603633 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:53:56.604252 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:56.604306 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:53:56.604332 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:56.604428 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:56.604548 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 06:53:56.604648 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260807 06:53:56.605798 24774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:56.698474 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 24621
I20260807 06:53:56.723604 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 24136
I20260807 06:53:56.724774 24219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:56.838519 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 24136
2026-08-07T06:53:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3022 ms)
[----------] 6 tests from EncryptionPolicyTest (24659 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:53:56 dist-test-slave-3kk6 krb5kdc[24853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:53:56 dist-test-slave-3kk6 krb5kdc[24853](info): set up 2 sockets
Aug 07 06:53:56 dist-test-slave-3kk6 krb5kdc[24853](info): commencing operation
krb5kdc: starting...
W20260807 06:53:58.182406 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:53:58 dist-test-slave-3kk6 krb5kdc[24853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:53:58.572538 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:53:58.942317 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:53:58.942723 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:53:58.942862 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:53:58.950500 24870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:53:58.950639 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:53:58.950767 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:53:58.950841 24870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:53:58.950896 24870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:53:58.961438 24870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:53:58.962854 24870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:53:58.964214 24870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:53:58.977001 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:58.977025 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:53:58.977061 24879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:53:58.978505 24870 server_base.cc:1034] running on GCE node
I20260807 06:53:58.979593 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:53:58.981225 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:53:58.982502 24870 hybrid_clock.cc:648] HybridClock initialized: now 1786085638982463 us; error 72 us; skew 500 ppm
I20260807 06:53:58.982966 24870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:53:58 dist-test-slave-3kk6 krb5kdc[24853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:53:58.986580 24870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:53:58.988440 24870 webserver.cc:460] Webserver started at http://127.0.0.1:46353/ using document root <none> and password file <none>
I20260807 06:53:58.989218 24870 fs_manager.cc:359] Metadata directory not provided
I20260807 06:53:58.989394 24870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:53:58.989967 24870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:53:58.993549 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "25066eff5b104df9892b0221221435ca"
format_stamp: "Formatted at 2026-08-07 06:53:58 on dist-test-slave-3kk6"
server_key: "53e2c93208dd9ed200e97054b4671af4"
server_key_iv: "743bc6fef1119d9b04830802d5ed4dc3"
server_key_version: "encryptionkey@0"
I20260807 06:53:58.994474 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "25066eff5b104df9892b0221221435ca"
format_stamp: "Formatted at 2026-08-07 06:53:58 on dist-test-slave-3kk6"
server_key: "53e2c93208dd9ed200e97054b4671af4"
server_key_iv: "743bc6fef1119d9b04830802d5ed4dc3"
server_key_version: "encryptionkey@0"
I20260807 06:53:59.000404 24870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 06:53:59.004391 24892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:53:59.005945 24870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:53:59.006165 24870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "25066eff5b104df9892b0221221435ca"
format_stamp: "Formatted at 2026-08-07 06:53:58 on dist-test-slave-3kk6"
server_key: "53e2c93208dd9ed200e97054b4671af4"
server_key_iv: "743bc6fef1119d9b04830802d5ed4dc3"
server_key_version: "encryptionkey@0"
I20260807 06:53:59.006404 24870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:53:59.024108 24870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:53:59.028753 24870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:53:59.029102 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:53:59.049243 24870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43587
I20260807 06:53:59.049305 24960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43587 every 8 connection(s)
I20260807 06:53:59.050879 24870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:53:59.051368 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 24870
I20260807 06:53:59.051642 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:53:59.052518 27865 external_mini_cluster.cc:1428] Setting key 79c8e31822f7b4f82ac35a7e9e4d30de
I20260807 06:53:59.055940 24961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:53:59 dist-test-slave-3kk6 krb5kdc[24853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:53:59.069514 24961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca: Bootstrap starting.
I20260807 06:53:59.073988 24961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca: Neither blocks nor log segments found. Creating new log.
I20260807 06:53:59.075335 24961 log.cc:824] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca: Log is configured to *not* fsync() on all Append() calls
I20260807 06:53:59.079087 24961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca: No bootstrap required, opened a new log
I20260807 06:53:59.079552 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:59.054676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42003 (local address 127.0.0.1:43587)
0807 06:53:59.055543 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:59.055559 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:59.055612 (+ 53us) server_negotiation.cc:407] Connection header received
0807 06:53:59.058015 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:59.058046 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:59.058573 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:59.059062 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:59.060562 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:59.062800 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:59.064182 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:59.065259 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:59.065457 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:59.070196 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:59.070208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:59.070229 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:59.070277 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:59.074540 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:59.075496 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:59.075513 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:59.075524 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:59.075749 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:59.076328 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:59.076332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:59.076334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:59.076994 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:59.077304 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:59.077779 (+ 475us) server_negotiation.cc:299] Negotiation successful
0807 06:53:59.078131 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":325,"threads_started":1}
I20260807 06:53:59.084090 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:53:59.088212 24961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25066eff5b104df9892b0221221435ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260807 06:53:59.088647 24961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:53:59.088771 24961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25066eff5b104df9892b0221221435ca, State: Initialized, Role: FOLLOWER
I20260807 06:53:59.089695 24961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25066eff5b104df9892b0221221435ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260807 06:53:59.090024 24961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:53:59.090189 24961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:53:59.090412 24961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:53:59.093541 24961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25066eff5b104df9892b0221221435ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260807 06:53:59.094129 24961 leader_election.cc:302] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25066eff5b104df9892b0221221435ca; no voters:
I20260807 06:53:59.094874 24961 leader_election.cc:288] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:53:59.095958 24973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:53:59.097548 24973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [term 1 LEADER]: Becoming Leader. State: Replica: 25066eff5b104df9892b0221221435ca, State: Running, Role: LEADER
I20260807 06:53:59.098371 24973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25066eff5b104df9892b0221221435ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260807 06:53:59.101137 24961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:53:59.101876 24974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25066eff5b104df9892b0221221435ca. Latest consensus state: current_term: 1 leader_uuid: "25066eff5b104df9892b0221221435ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25066eff5b104df9892b0221221435ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } } }
I20260807 06:53:59.102917 24974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:59.102531 24975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25066eff5b104df9892b0221221435ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25066eff5b104df9892b0221221435ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } } }
I20260807 06:53:59.103514 24975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca [sys.catalog]: This master's current role is: LEADER
I20260807 06:53:59.110090 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:53:59.089262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52355 (local address 127.0.0.1:43587)
0807 06:53:59.089462 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 06:53:59.089470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:53:59.089859 (+ 389us) server_negotiation.cc:407] Connection header received
0807 06:53:59.090103 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:53:59.090109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:53:59.090204 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:53:59.090327 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:53:59.092022 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:59.093276 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:59.094817 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:53:59.095454 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:53:59.095610 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:53:59.099083 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:53:59.099091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:53:59.099096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:53:59.099147 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:53:59.103780 (+ 4633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:59.105167 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:59.105177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:59.105184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:59.105422 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:53:59.106293 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:53:59.106301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:53:59.106307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:53:59.106899 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:53:59.107306 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0807 06:53:59.109170 (+ 1864us) server_negotiation.cc:299] Negotiation successful
0807 06:53:59.109470 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 06:53:59.112226 24978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:53:59.118557 24978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:53:59.128685 24978 catalog_manager.cc:1347] Generated new cluster ID: c756a641e2f84a75ba6a703cb115872f
I20260807 06:53:59.128845 24978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:53:59.142614 24978 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:53:59.144618 24978 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:53:59.161387 24978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25066eff5b104df9892b0221221435ca: Generated new TSK 0
I20260807 06:53:59.162595 24978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:53:59.174108 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 24870
I20260807 06:53:59.175949 24952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:53:59.271608 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 24870
W20260807 06:53:59.303128 24850 connection.cc:535] client connection to 127.0.0.1:43587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43587): Connection reset by peer (error 104)
2026-08-07T06:53:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:53:59 dist-test-slave-3kk6 krb5kdc[25004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:53:59 dist-test-slave-3kk6 krb5kdc[25004](info): set up 2 sockets
Aug 07 06:53:59 dist-test-slave-3kk6 krb5kdc[25004](info): commencing operation
krb5kdc: starting...
W20260807 06:54:00.633677 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.003s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:00 dist-test-slave-3kk6 krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:01.022478 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:01.403353 25021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:01.403698 25021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:01.403810 25021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:01.411504 25021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:01.411674 25021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:01.411725 25021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:01.411803 25021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:01.411850 25021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:01.422464 25021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:01.424018 25021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:01.425450 25021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:01.438762 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:01.438895 25030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:01.438791 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:01.439658 25021 server_base.cc:1034] running on GCE node
I20260807 06:54:01.440519 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:01.442149 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:01.443384 25021 hybrid_clock.cc:648] HybridClock initialized: now 1786085641443349 us; error 49 us; skew 500 ppm
I20260807 06:54:01.443778 25021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:01 dist-test-slave-3kk6 krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:01.451771 25021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:01.453589 25021 webserver.cc:460] Webserver started at http://127.0.0.1:38567/ using document root <none> and password file <none>
I20260807 06:54:01.454388 25021 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:01.454558 25021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:01.454944 25021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:01.458427 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "ee7d18ddc989486e920d26e20d9a4801"
format_stamp: "Formatted at 2026-08-07 06:54:01 on dist-test-slave-3kk6"
server_key: "a4c79bff85a10cb62b5db606c261c2d8"
server_key_iv: "146aef0fa813eafbd6fbdb042fa9c5d5"
server_key_version: "encryptionkey@0"
I20260807 06:54:01.459456 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "ee7d18ddc989486e920d26e20d9a4801"
format_stamp: "Formatted at 2026-08-07 06:54:01 on dist-test-slave-3kk6"
server_key: "a4c79bff85a10cb62b5db606c261c2d8"
server_key_iv: "146aef0fa813eafbd6fbdb042fa9c5d5"
server_key_version: "encryptionkey@0"
I20260807 06:54:01.465399 25021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:54:01.469540 25042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:01.471163 25021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:54:01.471400 25021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "ee7d18ddc989486e920d26e20d9a4801"
format_stamp: "Formatted at 2026-08-07 06:54:01 on dist-test-slave-3kk6"
server_key: "a4c79bff85a10cb62b5db606c261c2d8"
server_key_iv: "146aef0fa813eafbd6fbdb042fa9c5d5"
server_key_version: "encryptionkey@0"
I20260807 06:54:01.471648 25021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:01.485468 25021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:01.490304 25021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:01.490690 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:01.511500 25021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37203
I20260807 06:54:01.511575 25106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37203 every 8 connection(s)
I20260807 06:54:01.513221 25021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:01.517400 25108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:01.523108 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25021
I20260807 06:54:01.523329 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:01.523836 27865 external_mini_cluster.cc:1428] Setting key 8eedb1d5af8b269c01779c2ce84be8f2
I20260807 06:54:01.529562 25108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801: Bootstrap starting.
I20260807 06:54:01.533792 25108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:01.535202 25108 log.cc:824] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:01.538911 25108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801: No bootstrap required, opened a new log
I20260807 06:54:01.539470 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:01.525841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55965 (local address 127.0.0.1:37203)
0807 06:54:01.526853 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:01.526866 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:01.526912 (+ 46us) server_negotiation.cc:407] Connection header received
0807 06:54:01.528743 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:01.528769 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:01.529222 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:01.529614 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:01.530381 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:01.531860 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:01.534073 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:01.534936 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:01.535097 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:01.536745 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:01.536757 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:01.536778 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:01.537381 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:01.537463 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:01.537488 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:01.537608 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:01.538086 (+ 478us) server_negotiation.cc:299] Negotiation successful
0807 06:54:01.538288 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":468,"threads_started":1}
I20260807 06:54:01.543704 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:01.548069 25108 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7d18ddc989486e920d26e20d9a4801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260807 06:54:01.548594 25108 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:01.548741 25108 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7d18ddc989486e920d26e20d9a4801, State: Initialized, Role: FOLLOWER
I20260807 06:54:01.549606 25108 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7d18ddc989486e920d26e20d9a4801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260807 06:54:01.549969 25108 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:01.550158 25108 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:01.550364 25108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:01.553849 25108 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7d18ddc989486e920d26e20d9a4801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260807 06:54:01.554477 25108 leader_election.cc:302] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee7d18ddc989486e920d26e20d9a4801; no voters:
I20260807 06:54:01.555347 25108 leader_election.cc:288] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:01.555533 25121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:01.557672 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:01.548786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56387 (local address 127.0.0.1:37203)
0807 06:54:01.549019 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:01.549028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:01.549598 (+ 570us) server_negotiation.cc:407] Connection header received
0807 06:54:01.549859 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:01.549864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:01.549991 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:01.550122 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:01.550868 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:01.551754 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:01.553742 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:01.554314 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:01.554474 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:01.556009 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:01.556019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:01.556028 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:01.556065 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:01.556163 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:01.556201 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:01.556964 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:01.557079 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 06:54:01.557156 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:54:01.557869 25108 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:01.559079 25121 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [term 1 LEADER]: Becoming Leader. State: Replica: ee7d18ddc989486e920d26e20d9a4801, State: Running, Role: LEADER
I20260807 06:54:01.559827 25121 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7d18ddc989486e920d26e20d9a4801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260807 06:54:01.563230 25122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee7d18ddc989486e920d26e20d9a4801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7d18ddc989486e920d26e20d9a4801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } } }
I20260807 06:54:01.563634 25123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee7d18ddc989486e920d26e20d9a4801. Latest consensus state: current_term: 1 leader_uuid: "ee7d18ddc989486e920d26e20d9a4801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7d18ddc989486e920d26e20d9a4801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } } }
I20260807 06:54:01.563895 25122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:01.563941 25123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:01.564921 25131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:01.572373 25131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:01.586181 25131 catalog_manager.cc:1347] Generated new cluster ID: eee14229344544c290b99cf0aa583836
I20260807 06:54:01.586342 25131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:01.605700 25131 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:01.607213 25131 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:01.620579 25131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee7d18ddc989486e920d26e20d9a4801: Generated new TSK 0
I20260807 06:54:01.621557 25131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:01.631882 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25021
W20260807 06:54:01.633183 25102 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55965
W20260807 06:54:01.634153 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25021: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:01.634370 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25021
W20260807 06:54:01.659478 25001 connection.cc:535] client connection to 127.0.0.1:37203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37203): Connection reset by peer (error 104)
2026-08-07T06:54:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:01 dist-test-slave-3kk6 krb5kdc[25148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:01 dist-test-slave-3kk6 krb5kdc[25148](info): set up 2 sockets
Aug 07 06:54:01 dist-test-slave-3kk6 krb5kdc[25148](info): commencing operation
krb5kdc: starting...
W20260807 06:54:02.989616 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:03 dist-test-slave-3kk6 krb5kdc[25148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:03.352123 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:03.712137 25168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:03.712535 25168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:03.712724 25168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:03.720528 25168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:03.720818 25168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:03.720922 25168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:03.721027 25168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:03.721099 25168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:03.731465 25168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:03.732923 25168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:03.734324 25168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:03.746860 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:03.746882 25174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:03.746905 25176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:03.747828 25168 server_base.cc:1034] running on GCE node
I20260807 06:54:03.748711 25168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:03.750149 25168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:03.751363 25168 hybrid_clock.cc:648] HybridClock initialized: now 1786085643751302 us; error 62 us; skew 500 ppm
I20260807 06:54:03.751740 25168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:03 dist-test-slave-3kk6 krb5kdc[25148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:03.755098 25168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:03.756614 25168 webserver.cc:460] Webserver started at http://127.0.0.1:34823/ using document root <none> and password file <none>
I20260807 06:54:03.757340 25168 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:03.757490 25168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:03.757818 25168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:03.760982 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "8970f86801714980bb8d84225f02c3c1"
format_stamp: "Formatted at 2026-08-07 06:54:03 on dist-test-slave-3kk6"
server_key: "ee701e235d55cb083c8dfdc26fd0051e"
server_key_iv: "2b9eca3053698f1b04c9a0590e59e6ad"
server_key_version: "encryptionkey@0"
I20260807 06:54:03.761795 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "8970f86801714980bb8d84225f02c3c1"
format_stamp: "Formatted at 2026-08-07 06:54:03 on dist-test-slave-3kk6"
server_key: "ee701e235d55cb083c8dfdc26fd0051e"
server_key_iv: "2b9eca3053698f1b04c9a0590e59e6ad"
server_key_version: "encryptionkey@0"
I20260807 06:54:03.767277 25168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 06:54:03.770917 25190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:03.772212 25168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:03.772406 25168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "8970f86801714980bb8d84225f02c3c1"
format_stamp: "Formatted at 2026-08-07 06:54:03 on dist-test-slave-3kk6"
server_key: "ee701e235d55cb083c8dfdc26fd0051e"
server_key_iv: "2b9eca3053698f1b04c9a0590e59e6ad"
server_key_version: "encryptionkey@0"
I20260807 06:54:03.772614 25168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:03.778333 25168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:03.782549 25168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:03.782943 25168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:03.802309 25168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45087
I20260807 06:54:03.802343 25257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45087 every 8 connection(s)
I20260807 06:54:03.803905 25168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:03.807932 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25168
I20260807 06:54:03.808007 25258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:03.808145 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:03.808634 27865 external_mini_cluster.cc:1428] Setting key c45a3409777fe12216a7d7e845fa2f34
I20260807 06:54:03.817775 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:03.810636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38821 (local address 127.0.0.1:45087)
0807 06:54:03.811665 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:03.811681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:03.811740 (+ 59us) server_negotiation.cc:407] Connection header received
0807 06:54:03.813651 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:03.813693 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:03.814191 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:03.814708 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:03.815751 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:03.815759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:03.815778 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:03.815806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:03.815889 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:03.815922 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:03.816154 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:03.816305 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 06:54:03.816514 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":413,"threads_started":1}
I20260807 06:54:03.821784 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:03.823644 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1: Bootstrap starting.
I20260807 06:54:03.827165 25258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:03.828439 25258 log.cc:824] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:03.829186 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:03.826535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59533 (local address 127.0.0.1:45087)
0807 06:54:03.826749 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:03.826757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:03.827109 (+ 352us) server_negotiation.cc:407] Connection header received
0807 06:54:03.827323 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:03.827334 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:03.827387 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:03.827493 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:03.828496 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:03.828501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:03.828504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:03.828519 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:03.828557 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:03.828571 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:03.828700 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:03.828852 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 06:54:03.828915 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 06:54:03.831768 25258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1: No bootstrap required, opened a new log
I20260807 06:54:03.839457 25258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8970f86801714980bb8d84225f02c3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260807 06:54:03.839862 25258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:03.839989 25258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8970f86801714980bb8d84225f02c3c1, State: Initialized, Role: FOLLOWER
I20260807 06:54:03.840569 25258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8970f86801714980bb8d84225f02c3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260807 06:54:03.840812 25258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:03.840981 25258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:03.841162 25258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:03.843513 25258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8970f86801714980bb8d84225f02c3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260807 06:54:03.844031 25258 leader_election.cc:302] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8970f86801714980bb8d84225f02c3c1; no voters:
I20260807 06:54:03.844790 25258 leader_election.cc:288] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:03.844949 25271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:03.845701 25271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [term 1 LEADER]: Becoming Leader. State: Replica: 8970f86801714980bb8d84225f02c3c1, State: Running, Role: LEADER
I20260807 06:54:03.846269 25271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8970f86801714980bb8d84225f02c3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260807 06:54:03.846992 25258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:03.850374 25274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8970f86801714980bb8d84225f02c3c1. Latest consensus state: current_term: 1 leader_uuid: "8970f86801714980bb8d84225f02c3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8970f86801714980bb8d84225f02c3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } } }
I20260807 06:54:03.851532 25274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:03.850462 25273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8970f86801714980bb8d84225f02c3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8970f86801714980bb8d84225f02c3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } } }
I20260807 06:54:03.851727 25273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:03.855180 25283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:03.860261 25283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:03.869943 25283 catalog_manager.cc:1347] Generated new cluster ID: d0c1aee64dd94eb8b86d7e82740758d2
I20260807 06:54:03.870071 25283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:03.885843 25283 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:03.886355 25283 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:03.900120 25283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8970f86801714980bb8d84225f02c3c1: Generated new TSK 0
I20260807 06:54:03.901207 25283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:03.921240 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25168
W20260807 06:54:03.922435 25252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38821
W20260807 06:54:03.923238 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25168: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:03.923394 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25168
2026-08-07T06:54:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:04 dist-test-slave-3kk6 krb5kdc[25299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:04 dist-test-slave-3kk6 krb5kdc[25299](info): set up 2 sockets
Aug 07 06:54:04 dist-test-slave-3kk6 krb5kdc[25299](info): commencing operation
krb5kdc: starting...
W20260807 06:54:05.261104 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.003s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:05 dist-test-slave-3kk6 krb5kdc[25299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:05.629904 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:05.996008 25316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:05.996421 25316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:05.996608 25316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:06.004103 25316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:06.004285 25316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:06.004402 25316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:06.004518 25316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:06.004607 25316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:06.014885 25316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:06.016389 25316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:06.017786 25316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:06.030725 25323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:06.030725 25328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:06.031453 25326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:06.032070 25316 server_base.cc:1034] running on GCE node
I20260807 06:54:06.032965 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:06.034453 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:06.035702 25316 hybrid_clock.cc:648] HybridClock initialized: now 1786085646035656 us; error 59 us; skew 500 ppm
I20260807 06:54:06.036108 25316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:06 dist-test-slave-3kk6 krb5kdc[25299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:06.039428 25316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:06.041030 25316 webserver.cc:460] Webserver started at http://127.0.0.1:44833/ using document root <none> and password file <none>
I20260807 06:54:06.041739 25316 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:06.041911 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:06.042241 25316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:06.045445 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "a5672ba53e034e61a155fb88bdc7706a"
format_stamp: "Formatted at 2026-08-07 06:54:06 on dist-test-slave-3kk6"
server_key: "137a58d4e603eaab315d40c6cdb6a23c"
server_key_iv: "22bdac140a8dba8b0b304c759192e024"
server_key_version: "encryptionkey@0"
I20260807 06:54:06.046291 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "a5672ba53e034e61a155fb88bdc7706a"
format_stamp: "Formatted at 2026-08-07 06:54:06 on dist-test-slave-3kk6"
server_key: "137a58d4e603eaab315d40c6cdb6a23c"
server_key_iv: "22bdac140a8dba8b0b304c759192e024"
server_key_version: "encryptionkey@0"
I20260807 06:54:06.051729 25316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 06:54:06.055420 25336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:06.056787 25316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:06.057003 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "a5672ba53e034e61a155fb88bdc7706a"
format_stamp: "Formatted at 2026-08-07 06:54:06 on dist-test-slave-3kk6"
server_key: "137a58d4e603eaab315d40c6cdb6a23c"
server_key_iv: "22bdac140a8dba8b0b304c759192e024"
server_key_version: "encryptionkey@0"
I20260807 06:54:06.057224 25316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:06.082123 25316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:06.086391 25316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:06.086748 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:06.109064 25316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33369
I20260807 06:54:06.109091 25407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33369 every 8 connection(s)
I20260807 06:54:06.110745 25316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:06.114567 25408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:06.120958 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25316
I20260807 06:54:06.121173 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:06.121758 27865 external_mini_cluster.cc:1428] Setting key 395072fecc29c0811b776aece79c8816
I20260807 06:54:06.126070 25408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a: Bootstrap starting.
I20260807 06:54:06.130014 25408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:06.131261 25408 log.cc:824] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:06.134613 25408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a: No bootstrap required, opened a new log
Aug 07 06:54:06 dist-test-slave-3kk6 krb5kdc[25299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:54:06.143606 25408 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5672ba53e034e61a155fb88bdc7706a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260807 06:54:06.144032 25408 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:06.144165 25408 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5672ba53e034e61a155fb88bdc7706a, State: Initialized, Role: FOLLOWER
I20260807 06:54:06.144853 25408 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5672ba53e034e61a155fb88bdc7706a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260807 06:54:06.145129 25408 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:06.145296 25408 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:06.145502 25408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:06.146860 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:06.123773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46829 (local address 127.0.0.1:33369)
0807 06:54:06.124670 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:06.124682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:06.124725 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:54:06.126514 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:06.126540 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:06.126999 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:06.127333 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:06.129383 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:06.130889 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:06.132703 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:06.133428 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:06.133570 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:06.138552 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:06.138561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:06.138579 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:06.138621 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:06.142003 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:06.142904 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:06.142911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:06.142919 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:06.143113 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:06.143542 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:06.143546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:06.143548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:06.144166 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:06.144487 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:06.144972 (+ 485us) server_negotiation.cc:299] Negotiation successful
0807 06:54:06.145685 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":446,"threads_started":1}
I20260807 06:54:06.148180 25408 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5672ba53e034e61a155fb88bdc7706a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260807 06:54:06.148710 25408 leader_election.cc:302] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5672ba53e034e61a155fb88bdc7706a; no voters:
I20260807 06:54:06.149457 25408 leader_election.cc:288] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:06.149580 25414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:06.150874 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:06.150779 25414 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [term 1 LEADER]: Becoming Leader. State: Replica: a5672ba53e034e61a155fb88bdc7706a, State: Running, Role: LEADER
I20260807 06:54:06.151407 25414 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5672ba53e034e61a155fb88bdc7706a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260807 06:54:06.151964 25408 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:06.154589 25415 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5672ba53e034e61a155fb88bdc7706a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5672ba53e034e61a155fb88bdc7706a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } } }
I20260807 06:54:06.155112 25415 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:06.156106 25416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5672ba53e034e61a155fb88bdc7706a. Latest consensus state: current_term: 1 leader_uuid: "a5672ba53e034e61a155fb88bdc7706a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5672ba53e034e61a155fb88bdc7706a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } } }
I20260807 06:54:06.156612 25416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:06.159089 25420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:06.166263 25420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:06.179390 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:06.160576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41477 (local address 127.0.0.1:33369)
0807 06:54:06.161423 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:06.161429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:06.161453 (+ 24us) server_negotiation.cc:407] Connection header received
0807 06:54:06.161684 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:06.161690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:06.161813 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:06.161921 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:06.164314 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:06.165542 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:06.167528 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:06.168239 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:06.168345 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:06.171800 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:06.171807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:06.171812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:06.171862 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:06.175735 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:06.176666 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:06.176671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:06.176674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:06.176865 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:06.177387 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:06.177393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:06.177397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:06.177857 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:06.178097 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:06.178774 (+ 677us) server_negotiation.cc:299] Negotiation successful
0807 06:54:06.179015 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695}
I20260807 06:54:06.189165 25420 catalog_manager.cc:1347] Generated new cluster ID: 606075b432ab45329a10c263e2224233
I20260807 06:54:06.189575 25420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:06.228001 25420 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:06.229914 25420 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:06.243237 25420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5672ba53e034e61a155fb88bdc7706a: Generated new TSK 0
I20260807 06:54:06.244300 25420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:06.251312 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25316
I20260807 06:54:06.252629 25402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:06.348253 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25316
2026-08-07T06:54:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:06 dist-test-slave-3kk6 krb5kdc[25444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:06 dist-test-slave-3kk6 krb5kdc[25444](info): set up 2 sockets
Aug 07 06:54:06 dist-test-slave-3kk6 krb5kdc[25444](info): commencing operation
krb5kdc: starting...
W20260807 06:54:07.680691 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:07 dist-test-slave-3kk6 krb5kdc[25444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:08.054064 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:08.424849 25459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:08.425184 25459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:08.425282 25459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:08.433362 25459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:08.433498 25459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:08.433547 25459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:08.433611 25459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:08.433656 25459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:08.443928 25459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:08.445242 25459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:08.446593 25459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:08.459085 25470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:08.459118 25466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:08.459086 25467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:08.460613 25459 server_base.cc:1034] running on GCE node
I20260807 06:54:08.461553 25459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:08.463181 25459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:08.464437 25459 hybrid_clock.cc:648] HybridClock initialized: now 1786085648464400 us; error 48 us; skew 500 ppm
I20260807 06:54:08.464830 25459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:08 dist-test-slave-3kk6 krb5kdc[25444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:08.473626 25459 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:08.475423 25459 webserver.cc:460] Webserver started at http://127.0.0.1:46607/ using document root <none> and password file <none>
I20260807 06:54:08.476176 25459 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:08.476341 25459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:08.476680 25459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:08.480012 25459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "afcd3eb297fe43dbaea28f36392ff466"
format_stamp: "Formatted at 2026-08-07 06:54:08 on dist-test-slave-3kk6"
server_key: "a74a9c2220e10e8f89e98e2e8f6b2375"
server_key_iv: "5f17c28b2b22499f3dc9f2b04bf3c5fd"
server_key_version: "encryptionkey@0"
I20260807 06:54:08.480861 25459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "afcd3eb297fe43dbaea28f36392ff466"
format_stamp: "Formatted at 2026-08-07 06:54:08 on dist-test-slave-3kk6"
server_key: "a74a9c2220e10e8f89e98e2e8f6b2375"
server_key_iv: "5f17c28b2b22499f3dc9f2b04bf3c5fd"
server_key_version: "encryptionkey@0"
I20260807 06:54:08.486361 25459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 06:54:08.490160 25478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:08.491819 25459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:54:08.492029 25459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "afcd3eb297fe43dbaea28f36392ff466"
format_stamp: "Formatted at 2026-08-07 06:54:08 on dist-test-slave-3kk6"
server_key: "a74a9c2220e10e8f89e98e2e8f6b2375"
server_key_iv: "5f17c28b2b22499f3dc9f2b04bf3c5fd"
server_key_version: "encryptionkey@0"
I20260807 06:54:08.492259 25459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:08.522084 25459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:08.526963 25459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:08.527387 25459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:08.548734 25459 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36521
I20260807 06:54:08.548833 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36521 every 8 connection(s)
I20260807 06:54:08.550230 25459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:08.552335 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25459
I20260807 06:54:08.552556 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:08.553017 27865 external_mini_cluster.cc:1428] Setting key 8d60b6080acb24a5a3c3a404a541095f
I20260807 06:54:08.555052 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:08.570262 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:08.555075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37745 (local address 127.0.0.1:36521)
0807 06:54:08.556051 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:08.556068 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:08.556128 (+ 60us) server_negotiation.cc:407] Connection header received
0807 06:54:08.558642 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:08.558700 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:08.559289 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:08.559802 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:08.560444 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:08.562780 (+ 2336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:08.564277 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:08.565429 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:08.565634 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:08.567041 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:08.567052 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:08.567072 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:08.567947 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:08.568059 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:08.568101 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:08.568270 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:08.568463 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 06:54:08.568799 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":374,"threads_started":1}
I20260807 06:54:08.571851 25562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466: Bootstrap starting.
I20260807 06:54:08.574203 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:08.575881 25562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:08.577291 25562 log.cc:824] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:08.581478 25562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466: No bootstrap required, opened a new log
I20260807 06:54:08.594522 25562 raft_consensus.cc:348] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcd3eb297fe43dbaea28f36392ff466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36521 } }
I20260807 06:54:08.595139 25562 raft_consensus.cc:374] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:08.595261 25562 raft_consensus.cc:729] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcd3eb297fe43dbaea28f36392ff466, State: Initialized, Role: FOLLOWER
I20260807 06:54:08.595980 25562 consensus_queue.cc:260] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcd3eb297fe43dbaea28f36392ff466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36521 } }
I20260807 06:54:08.596369 25562 raft_consensus.cc:388] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:08.596558 25562 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:08.596820 25562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:08.599628 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:08.590706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48295 (local address 127.0.0.1:36521)
0807 06:54:08.590924 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:08.590934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:08.591490 (+ 556us) server_negotiation.cc:407] Connection header received
0807 06:54:08.591878 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:08.591891 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:08.592017 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:08.592164 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:08.592851 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:08.593968 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:08.596007 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:08.596938 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:08.597209 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:08.598682 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:08.598690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:08.598695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:08.598725 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:08.598782 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:08.598806 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:08.598927 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:08.599085 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 06:54:08.599211 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 06:54:08.600381 25562 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcd3eb297fe43dbaea28f36392ff466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36521 } }
I20260807 06:54:08.601030 25562 leader_election.cc:302] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afcd3eb297fe43dbaea28f36392ff466; no voters:
I20260807 06:54:08.601887 25562 leader_election.cc:288] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:08.602020 25573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:08.603305 25573 raft_consensus.cc:686] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [term 1 LEADER]: Becoming Leader. State: Replica: afcd3eb297fe43dbaea28f36392ff466, State: Running, Role: LEADER
I20260807 06:54:08.603994 25573 consensus_queue.cc:237] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcd3eb297fe43dbaea28f36392ff466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36521 } }
I20260807 06:54:08.604910 25562 sys_catalog.cc:564] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:08.607455 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcd3eb297fe43dbaea28f36392ff466. Latest consensus state: current_term: 1 leader_uuid: "afcd3eb297fe43dbaea28f36392ff466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcd3eb297fe43dbaea28f36392ff466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36521 } } }
I20260807 06:54:08.607908 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:08.608459 25574 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afcd3eb297fe43dbaea28f36392ff466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcd3eb297fe43dbaea28f36392ff466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36521 } } }
I20260807 06:54:08.608810 25574 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:08.608989 25582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:08.618907 25582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:08.632357 25582 catalog_manager.cc:1347] Generated new cluster ID: 7d1f0dee37df439ca5be0e0cca819670
I20260807 06:54:08.632548 25582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:08.643602 25582 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:08.645789 25582 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:08.659834 25582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afcd3eb297fe43dbaea28f36392ff466: Generated new TSK 0
I20260807 06:54:08.661070 25582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:08.672756 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25459
W20260807 06:54:08.673929 25556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37745
W20260807 06:54:08.674760 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25459: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:08.674911 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25459
2026-08-07T06:54:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:08 dist-test-slave-3kk6 krb5kdc[25599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:08 dist-test-slave-3kk6 krb5kdc[25599](info): set up 2 sockets
Aug 07 06:54:08 dist-test-slave-3kk6 krb5kdc[25599](info): commencing operation
krb5kdc: starting...
W20260807 06:54:10.021627 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:10 dist-test-slave-3kk6 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:10.396076 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:10.761971 25615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:10.762296 25615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:10.762494 25615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:10.770038 25615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:10.770179 25615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:10.770228 25615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:10.770319 25615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:10.770393 25615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:10.780819 25615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:10.782255 25615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:10.783690 25615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:10.796351 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:10.796334 25628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:10.796378 25625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:10.797361 25615 server_base.cc:1034] running on GCE node
I20260807 06:54:10.798205 25615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:10.799777 25615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:10.800992 25615 hybrid_clock.cc:648] HybridClock initialized: now 1786085650800950 us; error 32 us; skew 500 ppm
I20260807 06:54:10.801365 25615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:10 dist-test-slave-3kk6 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:10.804842 25615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:10.806432 25615 webserver.cc:460] Webserver started at http://127.0.0.1:42655/ using document root <none> and password file <none>
I20260807 06:54:10.807209 25615 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:10.807353 25615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:10.807699 25615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:10.810976 25615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f"
format_stamp: "Formatted at 2026-08-07 06:54:10 on dist-test-slave-3kk6"
server_key: "061a1566b38e154a3de5ac9e61afa8d9"
server_key_iv: "6de841772d66e9ecd2a55226b2784901"
server_key_version: "encryptionkey@0"
I20260807 06:54:10.811851 25615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f"
format_stamp: "Formatted at 2026-08-07 06:54:10 on dist-test-slave-3kk6"
server_key: "061a1566b38e154a3de5ac9e61afa8d9"
server_key_iv: "6de841772d66e9ecd2a55226b2784901"
server_key_version: "encryptionkey@0"
I20260807 06:54:10.817445 25615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 06:54:10.821347 25637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:10.822956 25615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:54:10.823156 25615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f"
format_stamp: "Formatted at 2026-08-07 06:54:10 on dist-test-slave-3kk6"
server_key: "061a1566b38e154a3de5ac9e61afa8d9"
server_key_iv: "6de841772d66e9ecd2a55226b2784901"
server_key_version: "encryptionkey@0"
I20260807 06:54:10.823365 25615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:10.829494 25615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:10.833909 25615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:10.834245 25615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:10.854315 25615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42389
I20260807 06:54:10.854362 25710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42389 every 8 connection(s)
I20260807 06:54:10.855921 25615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:10.859903 25711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:10.863976 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25615
I20260807 06:54:10.864200 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:10.864776 27865 external_mini_cluster.cc:1428] Setting key 2c303f4c99a43f6017cf86b44b8582f3
I20260807 06:54:10.873040 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:10.866880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34579 (local address 127.0.0.1:42389)
0807 06:54:10.867707 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:10.867719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:10.867760 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:54:10.869433 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:10.869456 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:10.869804 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:10.870228 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:10.871416 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:10.871422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:10.871435 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:10.871449 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:10.871509 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:10.871528 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:10.871783 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:10.871931 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 06:54:10.872119 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":375,"threads_started":1}
I20260807 06:54:10.873342 25711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f: Bootstrap starting.
I20260807 06:54:10.876920 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:10.877133 25711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:10.878166 25711 log.cc:824] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:10.880954 25711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f: No bootstrap required, opened a new log
I20260807 06:54:10.884418 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:10.881707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39879 (local address 127.0.0.1:42389)
0807 06:54:10.881956 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:10.881963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:10.882333 (+ 370us) server_negotiation.cc:407] Connection header received
0807 06:54:10.882516 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:10.882520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:10.882565 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:10.882695 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:10.883707 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:10.883712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:10.883716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:10.883729 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:10.883765 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:10.883783 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:10.883883 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:10.884020 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 06:54:10.884066 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:54:10.889534 25711 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260807 06:54:10.889938 25711 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:10.890056 25711 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa27c3ac297949b5ae0d6e7cda88ac0f, State: Initialized, Role: FOLLOWER
I20260807 06:54:10.890750 25711 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260807 06:54:10.890992 25711 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:10.891150 25711 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:10.891328 25711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:10.893612 25711 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260807 06:54:10.894156 25711 leader_election.cc:302] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa27c3ac297949b5ae0d6e7cda88ac0f; no voters:
I20260807 06:54:10.894789 25711 leader_election.cc:288] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:10.894940 25725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:10.895705 25725 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [term 1 LEADER]: Becoming Leader. State: Replica: aa27c3ac297949b5ae0d6e7cda88ac0f, State: Running, Role: LEADER
I20260807 06:54:10.896318 25725 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260807 06:54:10.897058 25711 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:10.899451 25726 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } } }
I20260807 06:54:10.899539 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa27c3ac297949b5ae0d6e7cda88ac0f. Latest consensus state: current_term: 1 leader_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa27c3ac297949b5ae0d6e7cda88ac0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } } }
I20260807 06:54:10.899860 25726 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:10.899945 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:10.902616 25732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:10.908947 25732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:10.925804 25732 catalog_manager.cc:1347] Generated new cluster ID: e8fdbab235c34defa9dd20cb4346e172
I20260807 06:54:10.926010 25732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:10.940593 25732 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:10.941181 25732 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:10.960427 25732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa27c3ac297949b5ae0d6e7cda88ac0f: Generated new TSK 0
I20260807 06:54:10.961395 25732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:10.972095 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25615
W20260807 06:54:10.973380 25703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34579
W20260807 06:54:10.974249 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:10.974438 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25615
2026-08-07T06:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:11 dist-test-slave-3kk6 krb5kdc[25756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:11 dist-test-slave-3kk6 krb5kdc[25756](info): set up 2 sockets
Aug 07 06:54:11 dist-test-slave-3kk6 krb5kdc[25756](info): commencing operation
krb5kdc: starting...
W20260807 06:54:12.378494 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.353s user 0.000s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:12 dist-test-slave-3kk6 krb5kdc[25756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:12.756560 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:13.124369 25774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:13.124733 25774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:13.124922 25774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:13.134284 25774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:13.134400 25774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:13.134505 25774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:13.134613 25774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:13.134706 25774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:13.145056 25774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:13.146487 25774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:13.147900 25774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:13.160857 25780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:13.160971 25781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:13.160948 25783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:13.161926 25774 server_base.cc:1034] running on GCE node
I20260807 06:54:13.162973 25774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:13.164805 25774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:13.166086 25774 hybrid_clock.cc:648] HybridClock initialized: now 1786085653166051 us; error 53 us; skew 500 ppm
I20260807 06:54:13.166531 25774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:13 dist-test-slave-3kk6 krb5kdc[25756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:13.170455 25774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:13.172425 25774 webserver.cc:460] Webserver started at http://127.0.0.1:34755/ using document root <none> and password file <none>
I20260807 06:54:13.173262 25774 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:13.173440 25774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:13.173815 25774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:13.177476 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "a7b8806871194961a0486aa85e73317f"
format_stamp: "Formatted at 2026-08-07 06:54:13 on dist-test-slave-3kk6"
server_key: "4f74ba291b6857e1cd36e3d6bc7c6aa3"
server_key_iv: "c5c5b4c7874a5d064423aff5252d3bb6"
server_key_version: "encryptionkey@0"
I20260807 06:54:13.178719 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "a7b8806871194961a0486aa85e73317f"
format_stamp: "Formatted at 2026-08-07 06:54:13 on dist-test-slave-3kk6"
server_key: "4f74ba291b6857e1cd36e3d6bc7c6aa3"
server_key_iv: "c5c5b4c7874a5d064423aff5252d3bb6"
server_key_version: "encryptionkey@0"
I20260807 06:54:13.184778 25774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260807 06:54:13.188771 25792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:13.190220 25774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 06:54:13.190428 25774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "a7b8806871194961a0486aa85e73317f"
format_stamp: "Formatted at 2026-08-07 06:54:13 on dist-test-slave-3kk6"
server_key: "4f74ba291b6857e1cd36e3d6bc7c6aa3"
server_key_iv: "c5c5b4c7874a5d064423aff5252d3bb6"
server_key_version: "encryptionkey@0"
I20260807 06:54:13.190690 25774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:13.207376 25774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:13.213902 25774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:13.214274 25774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:13.234495 25774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35959
I20260807 06:54:13.234539 25861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35959 every 8 connection(s)
I20260807 06:54:13.236089 25774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:13.240217 25862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:13.245826 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25774
I20260807 06:54:13.246063 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:13.246673 27865 external_mini_cluster.cc:1428] Setting key 655e900331427dcbe71cc9fc96564089
I20260807 06:54:13.252861 25862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f: Bootstrap starting.
I20260807 06:54:13.256276 25862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:13.257409 25862 log.cc:824] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:13.261492 25862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f: No bootstrap required, opened a new log
Aug 07 06:54:13 dist-test-slave-3kk6 krb5kdc[25756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:54:13.269943 25862 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8806871194961a0486aa85e73317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35959 } }
I20260807 06:54:13.270376 25862 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:13.270539 25862 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b8806871194961a0486aa85e73317f, State: Initialized, Role: FOLLOWER
I20260807 06:54:13.271356 25862 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8806871194961a0486aa85e73317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35959 } }
I20260807 06:54:13.271682 25862 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:13.271871 25862 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:13.272078 25862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:13.273299 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:13.248910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:59653 (local address 127.0.0.1:35959)
0807 06:54:13.249853 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:13.249865 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:13.249917 (+ 52us) server_negotiation.cc:407] Connection header received
0807 06:54:13.251845 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:13.251870 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:13.252301 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:13.252746 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:13.254429 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:13.255881 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:13.257363 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:13.258404 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:13.258694 (+ 290us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:13.264040 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:13.264050 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:13.264069 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:13.264113 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:13.267770 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:13.268897 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:13.268904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:13.268912 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:13.269115 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:13.269651 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:13.269655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:13.269657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:13.270273 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:13.270715 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:13.271275 (+ 560us) server_negotiation.cc:299] Negotiation successful
0807 06:54:13.272073 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":402,"threads_started":1}
I20260807 06:54:13.274941 25862 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8806871194961a0486aa85e73317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35959 } }
I20260807 06:54:13.275611 25862 leader_election.cc:302] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b8806871194961a0486aa85e73317f; no voters:
I20260807 06:54:13.276386 25862 leader_election.cc:288] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:13.276575 25877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:13.277813 25877 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [term 1 LEADER]: Becoming Leader. State: Replica: a7b8806871194961a0486aa85e73317f, State: Running, Role: LEADER
I20260807 06:54:13.278505 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:13.278486 25877 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8806871194961a0486aa85e73317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35959 } }
I20260807 06:54:13.279038 25862 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:13.281513 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b8806871194961a0486aa85e73317f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8806871194961a0486aa85e73317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35959 } } }
I20260807 06:54:13.281921 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:13.286855 25879 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b8806871194961a0486aa85e73317f. Latest consensus state: current_term: 1 leader_uuid: "a7b8806871194961a0486aa85e73317f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b8806871194961a0486aa85e73317f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35959 } } }
I20260807 06:54:13.287328 25879 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:13.292043 25882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:13.307425 25882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:13.322366 25882 catalog_manager.cc:1347] Generated new cluster ID: 64bd91167774413bae4db308c385b112
I20260807 06:54:13.322510 25882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:13.323537 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:13.294243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:56751 (local address 127.0.0.1:35959)
0807 06:54:13.294507 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:13.294516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:13.303882 (+ 9366us) server_negotiation.cc:407] Connection header received
0807 06:54:13.304228 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:13.304235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:13.304326 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:13.304421 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:13.306824 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:13.308084 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:13.310083 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:13.310829 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:13.310960 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:13.314889 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:13.314897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:13.314902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:13.314946 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:13.319195 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:13.320430 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:13.320436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:13.320439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:13.320620 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:13.321314 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:13.321319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:13.321322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:13.321796 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:13.322056 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:13.322897 (+ 841us) server_negotiation.cc:299] Negotiation successful
0807 06:54:13.323114 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:54:13.328073 25910 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:54:13.343741 25882 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:13.345621 25882 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:13.359248 25882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b8806871194961a0486aa85e73317f: Generated new TSK 0
I20260807 06:54:13.360271 25882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:13.370388 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25774
I20260807 06:54:13.372771 25856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:13.469071 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25774
W20260807 06:54:13.494498 25754 connection.cc:535] client connection to 127.0.0.1:35959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35959): Connection reset by peer (error 104)
2026-08-07T06:54:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:13 dist-test-slave-3kk6 krb5kdc[25917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:13 dist-test-slave-3kk6 krb5kdc[25917](info): set up 2 sockets
Aug 07 06:54:13 dist-test-slave-3kk6 krb5kdc[25917](info): commencing operation
krb5kdc: starting...
W20260807 06:54:14.838991 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.324s user 0.001s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:15 dist-test-slave-3kk6 krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:15.230007 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:15.610415 25937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:15.610870 25937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:15.610981 25937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:15.618613 25937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:15.618847 25937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:15.618904 25937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:15.618978 25937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:15.619024 25937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:15.629530 25937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:15.631042 25937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:15.632457 25937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:15.645494 25942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:15.645459 25947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:15.645534 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:15.646443 25937 server_base.cc:1034] running on GCE node
I20260807 06:54:15.647431 25937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:15.649142 25937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:15.650411 25937 hybrid_clock.cc:648] HybridClock initialized: now 1786085655650348 us; error 63 us; skew 500 ppm
I20260807 06:54:15.650863 25937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:15 dist-test-slave-3kk6 krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:15.654353 25937 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:15.656476 25937 webserver.cc:460] Webserver started at http://127.0.0.1:44787/ using document root <none> and password file <none>
I20260807 06:54:15.657289 25937 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:15.657452 25937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:15.657800 25937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:15.661171 25937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461"
format_stamp: "Formatted at 2026-08-07 06:54:15 on dist-test-slave-3kk6"
server_key: "6765eeeac416bfa29dcbbd30684d62a6"
server_key_iv: "4ae24c415e9ca3ce9017ba602561ab2a"
server_key_version: "encryptionkey@0"
I20260807 06:54:15.662003 25937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461"
format_stamp: "Formatted at 2026-08-07 06:54:15 on dist-test-slave-3kk6"
server_key: "6765eeeac416bfa29dcbbd30684d62a6"
server_key_iv: "4ae24c415e9ca3ce9017ba602561ab2a"
server_key_version: "encryptionkey@0"
I20260807 06:54:15.667680 25937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:54:15.671407 25957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:15.672853 25937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:15.673062 25937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461"
format_stamp: "Formatted at 2026-08-07 06:54:15 on dist-test-slave-3kk6"
server_key: "6765eeeac416bfa29dcbbd30684d62a6"
server_key_iv: "4ae24c415e9ca3ce9017ba602561ab2a"
server_key_version: "encryptionkey@0"
I20260807 06:54:15.673278 25937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:15.695568 25937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:15.700297 25937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:15.700636 25937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:15.720758 25937 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37357
I20260807 06:54:15.720803 26022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37357 every 8 connection(s)
I20260807 06:54:15.722275 25937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:15.726202 26023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:15.729741 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 25937
I20260807 06:54:15.729948 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:15.730546 27865 external_mini_cluster.cc:1428] Setting key 4d4fc4c0ee3c9588b7e1971a4267488c
I20260807 06:54:15.738907 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461: Bootstrap starting.
I20260807 06:54:15.742357 26023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:15.743664 26023 log.cc:824] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:15.744760 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:15.732701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:34111 (local address 127.0.0.1:37357)
0807 06:54:15.733497 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:15.733510 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:15.733552 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:54:15.735390 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:15.735415 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:15.735869 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:15.736304 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:15.736904 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:15.738228 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:15.739777 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:15.740487 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:15.740696 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:15.742083 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:15.742091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:15.742104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:15.742773 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:15.742859 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:15.742885 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:15.743060 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:15.743345 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 06:54:15.743507 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":324,"threads_started":1}
I20260807 06:54:15.747109 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461: No bootstrap required, opened a new log
I20260807 06:54:15.748888 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:15.755705 26023 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260807 06:54:15.756225 26023 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:15.756352 26023 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c037a3cc1d4accbdb9b64e9fc7a461, State: Initialized, Role: FOLLOWER
I20260807 06:54:15.757019 26023 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260807 06:54:15.757282 26023 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:15.757452 26023 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:15.757632 26023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:15.760916 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:15.753647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:44901 (local address 127.0.0.1:37357)
0807 06:54:15.753902 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:15.753909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:15.754288 (+ 379us) server_negotiation.cc:407] Connection header received
0807 06:54:15.754431 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:15.754437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:15.754524 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:15.754694 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:15.755219 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:15.756335 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:15.757933 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:15.758620 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:15.758860 (+ 240us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:15.760054 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:15.760060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:15.760065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:15.760088 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:15.760160 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:15.760185 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:15.760313 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:15.760437 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 06:54:15.760502 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 06:54:15.760872 26023 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260807 06:54:15.761459 26023 leader_election.cc:302] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3c037a3cc1d4accbdb9b64e9fc7a461; no voters:
I20260807 06:54:15.762164 26023 leader_election.cc:288] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:15.762315 26039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:15.763185 26039 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [term 1 LEADER]: Becoming Leader. State: Replica: b3c037a3cc1d4accbdb9b64e9fc7a461, State: Running, Role: LEADER
I20260807 06:54:15.763787 26039 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260807 06:54:15.764452 26023 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:15.767395 26040 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } } }
I20260807 06:54:15.767503 26041 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3c037a3cc1d4accbdb9b64e9fc7a461. Latest consensus state: current_term: 1 leader_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c037a3cc1d4accbdb9b64e9fc7a461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } } }
I20260807 06:54:15.767805 26040 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:15.767894 26041 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:15.770536 26045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:15.776345 26045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:15.786639 26045 catalog_manager.cc:1347] Generated new cluster ID: fbc43cd930864eca84109e0b825b0aa7
I20260807 06:54:15.786865 26045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:15.807911 26045 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:15.809371 26045 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:15.819875 26045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3c037a3cc1d4accbdb9b64e9fc7a461: Generated new TSK 0
I20260807 06:54:15.821059 26045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:15.833912 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25937
W20260807 06:54:15.835389 26018 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:34111
W20260807 06:54:15.836292 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 25937: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:15.836488 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 25937
W20260807 06:54:15.864102 25913 connection.cc:535] client connection to 127.0.0.1:37357 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37357): Connection reset by peer (error 104)
2026-08-07T06:54:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:15 dist-test-slave-3kk6 krb5kdc[26069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:15 dist-test-slave-3kk6 krb5kdc[26069](info): set up 2 sockets
Aug 07 06:54:15 dist-test-slave-3kk6 krb5kdc[26069](info): commencing operation
krb5kdc: starting...
W20260807 06:54:17.182368 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:17 dist-test-slave-3kk6 krb5kdc[26069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:17.558063 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:17.927237 26087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:17.927578 26087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:17.927675 26087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:17.935253 26087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:17.935402 26087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:17.935453 26087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:17.935518 26087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:17.935566 26087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:17.946033 26087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:17.947459 26087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:17.948809 26087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:17.962162 26095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:17.962070 26093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:17.962178 26092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:17.962975 26087 server_base.cc:1034] running on GCE node
I20260807 06:54:17.963897 26087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:17.965479 26087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:17.966766 26087 hybrid_clock.cc:648] HybridClock initialized: now 1786085657966711 us; error 40 us; skew 500 ppm
I20260807 06:54:17.967144 26087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:17 dist-test-slave-3kk6 krb5kdc[26069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:17.970530 26087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:17.972211 26087 webserver.cc:460] Webserver started at http://127.0.0.1:43833/ using document root <none> and password file <none>
I20260807 06:54:17.972981 26087 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:17.973142 26087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:17.973474 26087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:17.976802 26087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "90e5af5d5e3d4074a87ba285f9505c1e"
format_stamp: "Formatted at 2026-08-07 06:54:17 on dist-test-slave-3kk6"
server_key: "70b4f8f34a5fad54153104d01a6c2d7d"
server_key_iv: "7e32cab3ea8d761435b9e253f8748610"
server_key_version: "encryptionkey@0"
I20260807 06:54:17.977698 26087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "90e5af5d5e3d4074a87ba285f9505c1e"
format_stamp: "Formatted at 2026-08-07 06:54:17 on dist-test-slave-3kk6"
server_key: "70b4f8f34a5fad54153104d01a6c2d7d"
server_key_iv: "7e32cab3ea8d761435b9e253f8748610"
server_key_version: "encryptionkey@0"
I20260807 06:54:17.983757 26087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260807 06:54:17.987651 26108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:17.989218 26087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:17.989444 26087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "90e5af5d5e3d4074a87ba285f9505c1e"
format_stamp: "Formatted at 2026-08-07 06:54:17 on dist-test-slave-3kk6"
server_key: "70b4f8f34a5fad54153104d01a6c2d7d"
server_key_iv: "7e32cab3ea8d761435b9e253f8748610"
server_key_version: "encryptionkey@0"
I20260807 06:54:17.989679 26087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:17.996191 26087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:18.000784 26087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:18.001142 26087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:18.021025 26087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42115
I20260807 06:54:18.021071 26179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42115 every 8 connection(s)
I20260807 06:54:18.022572 26087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:18.026453 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:18.027066 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 26087
I20260807 06:54:18.027264 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:18.027711 27865 external_mini_cluster.cc:1428] Setting key 5a9ed2d96075877e3f1b2efa30460757
I20260807 06:54:18.036500 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:18.029597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:53889 (local address 127.0.0.1:42115)
0807 06:54:18.030511 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:18.030523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:18.030568 (+ 45us) server_negotiation.cc:407] Connection header received
0807 06:54:18.032755 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:18.032779 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:18.033146 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:18.033529 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:18.034830 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:18.034837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:18.034851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:18.034867 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:18.034932 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:18.034956 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:18.035159 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:18.035299 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 06:54:18.035511 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":443,"threads_started":1}
I20260807 06:54:18.040673 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:18.041126 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e: Bootstrap starting.
I20260807 06:54:18.044525 26180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:18.045877 26180 log.cc:824] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:18.048444 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:18.045655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:36877 (local address 127.0.0.1:42115)
0807 06:54:18.045868 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:18.045874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:18.046156 (+ 282us) server_negotiation.cc:407] Connection header received
0807 06:54:18.046365 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:18.046369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:18.046445 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:18.046580 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:18.047682 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:18.047687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:18.047691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:18.047711 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:18.047758 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:18.047777 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:18.047917 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:18.048094 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 06:54:18.048160 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 06:54:18.049638 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e: No bootstrap required, opened a new log
I20260807 06:54:18.057667 26180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42115 } }
I20260807 06:54:18.058127 26180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:18.058223 26180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e5af5d5e3d4074a87ba285f9505c1e, State: Initialized, Role: FOLLOWER
I20260807 06:54:18.059054 26180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42115 } }
I20260807 06:54:18.059322 26180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:18.059437 26180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:18.059572 26180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:18.062459 26180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42115 } }
I20260807 06:54:18.063086 26180 leader_election.cc:302] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e5af5d5e3d4074a87ba285f9505c1e; no voters:
I20260807 06:54:18.063796 26180 leader_election.cc:288] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:18.064004 26195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:18.064801 26195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [term 1 LEADER]: Becoming Leader. State: Replica: 90e5af5d5e3d4074a87ba285f9505c1e, State: Running, Role: LEADER
I20260807 06:54:18.065459 26195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42115 } }
I20260807 06:54:18.066118 26180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:18.068943 26196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42115 } } }
I20260807 06:54:18.069875 26196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:18.069921 26198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e5af5d5e3d4074a87ba285f9505c1e. Latest consensus state: current_term: 1 leader_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e5af5d5e3d4074a87ba285f9505c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42115 } } }
I20260807 06:54:18.070710 26198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:18.075719 26203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:18.082348 26203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:18.093979 26203 catalog_manager.cc:1347] Generated new cluster ID: 07723ad475fb4ed7806bf79e2527ac8c
I20260807 06:54:18.094128 26203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:18.117659 26203 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:18.118160 26203 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:18.138386 26203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e5af5d5e3d4074a87ba285f9505c1e: Generated new TSK 0
I20260807 06:54:18.139336 26203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:18.156673 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26087
W20260807 06:54:18.157930 26175 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:53889
W20260807 06:54:18.158836 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26087: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:18.159003 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 26087
2026-08-07T06:54:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:18 dist-test-slave-3kk6 krb5kdc[26230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:18 dist-test-slave-3kk6 krb5kdc[26230](info): set up 2 sockets
Aug 07 06:54:18 dist-test-slave-3kk6 krb5kdc[26230](info): commencing operation
krb5kdc: starting...
W20260807 06:54:19.522444 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.007s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:19 dist-test-slave-3kk6 krb5kdc[26230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:19.902268 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:20.273475 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:20.273804 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:20.273901 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:20.281201 26245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:20.281345 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:20.281394 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:20.281459 26245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:20.281505 26245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:20.292274 26245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:20.293664 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:20.295027 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:20.307215 26254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:20.307189 26252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:20.307588 26251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:20.308089 26245 server_base.cc:1034] running on GCE node
I20260807 06:54:20.308926 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:20.310437 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:20.311651 26245 hybrid_clock.cc:648] HybridClock initialized: now 1786085660311593 us; error 41 us; skew 500 ppm
I20260807 06:54:20.312044 26245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:20 dist-test-slave-3kk6 krb5kdc[26230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:20.315397 26245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:20.317037 26245 webserver.cc:460] Webserver started at http://127.0.0.1:44647/ using document root <none> and password file <none>
I20260807 06:54:20.317744 26245 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:20.317894 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:20.318241 26245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:20.321419 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "ec9114d5aa9044c4b513be0284516d9f"
format_stamp: "Formatted at 2026-08-07 06:54:20 on dist-test-slave-3kk6"
server_key: "398bfd5deccfb741cca83286e8d50a1f"
server_key_iv: "60380b430d6f4cb0018ede71e478563a"
server_key_version: "encryptionkey@0"
I20260807 06:54:20.322232 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "ec9114d5aa9044c4b513be0284516d9f"
format_stamp: "Formatted at 2026-08-07 06:54:20 on dist-test-slave-3kk6"
server_key: "398bfd5deccfb741cca83286e8d50a1f"
server_key_iv: "60380b430d6f4cb0018ede71e478563a"
server_key_version: "encryptionkey@0"
I20260807 06:54:20.327793 26245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 06:54:20.331653 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:20.333097 26245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 06:54:20.333290 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "ec9114d5aa9044c4b513be0284516d9f"
format_stamp: "Formatted at 2026-08-07 06:54:20 on dist-test-slave-3kk6"
server_key: "398bfd5deccfb741cca83286e8d50a1f"
server_key_iv: "60380b430d6f4cb0018ede71e478563a"
server_key_version: "encryptionkey@0"
I20260807 06:54:20.333518 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:20.356846 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:20.361683 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:20.362026 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:20.382014 26245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35249
I20260807 06:54:20.382047 26330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35249 every 8 connection(s)
I20260807 06:54:20.383553 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:20.388010 26332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:20.389952 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 26245
I20260807 06:54:20.390139 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:20.390584 27865 external_mini_cluster.cc:1428] Setting key 13a1d777c6e59d6be68218acc2ff2035
I20260807 06:54:20.403313 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f: Bootstrap starting.
Aug 07 06:54:20 dist-test-slave-3kk6 krb5kdc[26230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 06:54:20.407568 26332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:20.408747 26332 log.cc:824] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:20.411983 26332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f: No bootstrap required, opened a new log
I20260807 06:54:20.416203 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:20.392578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:42907 (local address 127.0.0.1:35249)
0807 06:54:20.393457 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:20.393470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:20.393519 (+ 49us) server_negotiation.cc:407] Connection header received
0807 06:54:20.395228 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:20.395253 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:20.395692 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:20.396049 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:20.398170 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:20.399745 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:20.401667 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:20.402388 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:20.402524 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:20.408087 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:20.408096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:20.408112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:20.408152 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:20.411499 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:20.412507 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:20.412514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:20.412520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:20.412692 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:20.413195 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:20.413199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:20.413201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:20.413780 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:20.414146 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:20.414756 (+ 610us) server_negotiation.cc:299] Negotiation successful
0807 06:54:20.415097 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":642,"thread_start_us":421,"threads_started":1}
I20260807 06:54:20.420361 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:20.420590 26332 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9114d5aa9044c4b513be0284516d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35249 } }
I20260807 06:54:20.420993 26332 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:20.421105 26332 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9114d5aa9044c4b513be0284516d9f, State: Initialized, Role: FOLLOWER
I20260807 06:54:20.421657 26332 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9114d5aa9044c4b513be0284516d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35249 } }
I20260807 06:54:20.421905 26332 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:20.422046 26332 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:20.422216 26332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:20.425045 26332 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9114d5aa9044c4b513be0284516d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35249 } }
I20260807 06:54:20.425715 26332 leader_election.cc:302] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9114d5aa9044c4b513be0284516d9f; no voters:
I20260807 06:54:20.426431 26332 leader_election.cc:288] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:20.426538 26348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:20.428683 26332 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:20.429272 26348 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [term 1 LEADER]: Becoming Leader. State: Replica: ec9114d5aa9044c4b513be0284516d9f, State: Running, Role: LEADER
I20260807 06:54:20.429863 26348 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9114d5aa9044c4b513be0284516d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35249 } }
I20260807 06:54:20.433254 26351 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9114d5aa9044c4b513be0284516d9f. Latest consensus state: current_term: 1 leader_uuid: "ec9114d5aa9044c4b513be0284516d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9114d5aa9044c4b513be0284516d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35249 } } }
I20260807 06:54:20.433673 26351 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:20.434235 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec9114d5aa9044c4b513be0284516d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9114d5aa9044c4b513be0284516d9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35249 } } }
I20260807 06:54:20.434520 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:20.445019 26361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:20.449616 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:20.425331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:55165 (local address 127.0.0.1:35249)
0807 06:54:20.425524 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:20.425531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:20.426021 (+ 490us) server_negotiation.cc:407] Connection header received
0807 06:54:20.426204 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:20.426214 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:20.426323 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:20.426462 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:20.428976 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:20.430353 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:20.432554 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:20.433486 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:20.433671 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:20.440066 (+ 6395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:20.440078 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:20.440086 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:20.440158 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:20.445353 (+ 5195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:20.446531 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:20.446540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:20.446546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:20.446770 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:20.447581 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:20.447592 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:20.447596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:20.448126 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:20.448431 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:20.448958 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 06:54:20.449189 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:54:20.451553 26361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:20.460997 26361 catalog_manager.cc:1347] Generated new cluster ID: d59857c189c9410bb804c3f0fdfe4eb1
I20260807 06:54:20.461143 26361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:20.477005 26361 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:20.479009 26361 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:20.497383 26361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec9114d5aa9044c4b513be0284516d9f: Generated new TSK 0
I20260807 06:54:20.498557 26361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:20.504663 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26245
I20260807 06:54:20.506765 26325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:20.601851 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 26245
W20260807 06:54:20.627949 26227 connection.cc:535] client connection to 127.0.0.1:35249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35249): Connection reset by peer (error 104)
2026-08-07T06:54:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:20 dist-test-slave-3kk6 krb5kdc[26382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:20 dist-test-slave-3kk6 krb5kdc[26382](info): set up 2 sockets
Aug 07 06:54:20 dist-test-slave-3kk6 krb5kdc[26382](info): commencing operation
krb5kdc: starting...
W20260807 06:54:21.973634 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:22 dist-test-slave-3kk6 krb5kdc[26382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:22.352135 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:22.718942 26403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:22.719345 26403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:22.719528 26403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:22.727869 26403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:22.728050 26403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:22.728152 26403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:22.728256 26403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:22.728331 26403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:22.739404 26403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:22.740869 26403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:22.742283 26403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:22.754952 26412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:22.754905 26414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:22.754976 26417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:22.756036 26403 server_base.cc:1034] running on GCE node
I20260807 06:54:22.756878 26403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:22.758524 26403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:22.759773 26403 hybrid_clock.cc:648] HybridClock initialized: now 1786085662759747 us; error 68 us; skew 500 ppm
I20260807 06:54:22.760164 26403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:22 dist-test-slave-3kk6 krb5kdc[26382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:22.763723 26403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:22.765443 26403 webserver.cc:460] Webserver started at http://127.0.0.1:44571/ using document root <none> and password file <none>
I20260807 06:54:22.766191 26403 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:22.766347 26403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:22.766726 26403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:22.769969 26403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "6956d7ccf0ab4c578b549dc85de1fc11"
format_stamp: "Formatted at 2026-08-07 06:54:22 on dist-test-slave-3kk6"
server_key: "374f9502423afd01cc842f72b5db8b4e"
server_key_iv: "25dbf7713b19c8799be1fbdfb72a697b"
server_key_version: "encryptionkey@0"
I20260807 06:54:22.770898 26403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "6956d7ccf0ab4c578b549dc85de1fc11"
format_stamp: "Formatted at 2026-08-07 06:54:22 on dist-test-slave-3kk6"
server_key: "374f9502423afd01cc842f72b5db8b4e"
server_key_iv: "25dbf7713b19c8799be1fbdfb72a697b"
server_key_version: "encryptionkey@0"
I20260807 06:54:22.776551 26403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:54:22.780411 26426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:22.781880 26403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:54:22.782080 26403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "6956d7ccf0ab4c578b549dc85de1fc11"
format_stamp: "Formatted at 2026-08-07 06:54:22 on dist-test-slave-3kk6"
server_key: "374f9502423afd01cc842f72b5db8b4e"
server_key_iv: "25dbf7713b19c8799be1fbdfb72a697b"
server_key_version: "encryptionkey@0"
I20260807 06:54:22.782295 26403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:22.804553 26403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:22.809358 26403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:22.809692 26403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:22.830000 26403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39091
I20260807 06:54:22.830047 26502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39091 every 8 connection(s)
I20260807 06:54:22.831518 26403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:22.835551 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:22.840381 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 26403
I20260807 06:54:22.840576 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:22.841181 27865 external_mini_cluster.cc:1428] Setting key 1d65bf286810d72be6ae05589ff1a164
I20260807 06:54:22.847862 26503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11: Bootstrap starting.
I20260807 06:54:22.851375 26503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:22.852517 26503 log.cc:824] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:22.856256 26503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11: No bootstrap required, opened a new log
I20260807 06:54:22.856278 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:22.843318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:52471 (local address 127.0.0.1:39091)
0807 06:54:22.844123 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:22.844135 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:22.844177 (+ 42us) server_negotiation.cc:407] Connection header received
0807 06:54:22.846398 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:22.846424 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:22.846995 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:22.847428 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:22.848044 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:22.849424 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:22.850969 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:22.851768 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:22.851987 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:22.853402 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:22.853410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:22.853424 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:22.854087 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:22.854194 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:22.854230 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:22.854407 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:22.854774 (+ 367us) server_negotiation.cc:299] Negotiation successful
0807 06:54:22.854977 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":341,"threads_started":1}
I20260807 06:54:22.860431 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:22.865010 26503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260807 06:54:22.865507 26503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:22.865636 26503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6956d7ccf0ab4c578b549dc85de1fc11, State: Initialized, Role: FOLLOWER
I20260807 06:54:22.866329 26503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260807 06:54:22.866603 26503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:22.866848 26503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:22.867085 26503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:22.869948 26503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260807 06:54:22.870518 26503 leader_election.cc:302] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6956d7ccf0ab4c578b549dc85de1fc11; no voters:
I20260807 06:54:22.871274 26503 leader_election.cc:288] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:22.871398 26515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:22.872467 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:22.864969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:57727 (local address 127.0.0.1:39091)
0807 06:54:22.865160 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:22.865166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:22.865563 (+ 397us) server_negotiation.cc:407] Connection header received
0807 06:54:22.865788 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:22.865793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:22.865875 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:22.865994 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:22.866565 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:22.867756 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:22.869129 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:22.869866 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:22.870019 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:22.871315 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:22.871326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:22.871329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:22.871348 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:22.871454 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:22.871475 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:22.871601 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:22.871958 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 06:54:22.872036 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:54:22.873471 26515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [term 1 LEADER]: Becoming Leader. State: Replica: 6956d7ccf0ab4c578b549dc85de1fc11, State: Running, Role: LEADER
I20260807 06:54:22.874147 26515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260807 06:54:22.874384 26503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:22.877413 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6956d7ccf0ab4c578b549dc85de1fc11. Latest consensus state: current_term: 1 leader_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } } }
I20260807 06:54:22.877933 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:22.878835 26516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6956d7ccf0ab4c578b549dc85de1fc11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } } }
I20260807 06:54:22.879175 26516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:22.880261 26525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:22.885169 26525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:22.894515 26525 catalog_manager.cc:1347] Generated new cluster ID: 257ac17a309f458b82d9bb6ee4352d16
I20260807 06:54:22.894883 26525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:22.905799 26525 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:22.907732 26525 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:22.918401 26525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6956d7ccf0ab4c578b549dc85de1fc11: Generated new TSK 0
I20260807 06:54:22.919567 26525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:22.927893 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26403
W20260807 06:54:22.929158 26492 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:52471
W20260807 06:54:22.930128 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26403: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:22.930291 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 26403
W20260807 06:54:22.959445 26380 connection.cc:535] client connection to 127.0.0.1:39091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39091): Connection reset by peer (error 104)
2026-08-07T06:54:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:23 dist-test-slave-3kk6 krb5kdc[26543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:23 dist-test-slave-3kk6 krb5kdc[26543](info): set up 2 sockets
Aug 07 06:54:23 dist-test-slave-3kk6 krb5kdc[26543](info): commencing operation
krb5kdc: starting...
W20260807 06:54:24.275693 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.001s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:24 dist-test-slave-3kk6 krb5kdc[26543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 06:54:24.650427 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:25.020605 26557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:25.020982 26557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:25.021155 26557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:25.028532 26557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:25.028688 26557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:25.028797 26557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:25.028913 26557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:25.028995 26557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:25.039558 26557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:25.041064 26557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:25.042490 26557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:25.055140 26562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:25.055231 26563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:25.055140 26566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:25.056100 26557 server_base.cc:1034] running on GCE node
I20260807 06:54:25.056957 26557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:25.058432 26557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:25.059633 26557 hybrid_clock.cc:648] HybridClock initialized: now 1786085665059568 us; error 60 us; skew 500 ppm
I20260807 06:54:25.059965 26557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:25 dist-test-slave-3kk6 krb5kdc[26543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:25.066196 26557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 06:54:25.067873 26557 webserver.cc:460] Webserver started at http://127.0.0.1:34549/ using document root <none> and password file <none>
I20260807 06:54:25.068609 26557 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:25.068749 26557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:25.069068 26557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:25.072340 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "aa0de3d7d9c5497fa6157632406aec8c"
format_stamp: "Formatted at 2026-08-07 06:54:25 on dist-test-slave-3kk6"
server_key: "3867f101ed961d276ae9b613e59f83db"
server_key_iv: "7700a7d8a01213de2a2c9919b12ea916"
server_key_version: "encryptionkey@0"
I20260807 06:54:25.073169 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "aa0de3d7d9c5497fa6157632406aec8c"
format_stamp: "Formatted at 2026-08-07 06:54:25 on dist-test-slave-3kk6"
server_key: "3867f101ed961d276ae9b613e59f83db"
server_key_iv: "7700a7d8a01213de2a2c9919b12ea916"
server_key_version: "encryptionkey@0"
I20260807 06:54:25.078641 26557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 06:54:25.082158 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:25.083586 26557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:54:25.083773 26557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "aa0de3d7d9c5497fa6157632406aec8c"
format_stamp: "Formatted at 2026-08-07 06:54:25 on dist-test-slave-3kk6"
server_key: "3867f101ed961d276ae9b613e59f83db"
server_key_iv: "7700a7d8a01213de2a2c9919b12ea916"
server_key_version: "encryptionkey@0"
I20260807 06:54:25.083976 26557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:25.089918 26557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:25.094435 26557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:25.094798 26557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:25.121513 26557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35537
I20260807 06:54:25.121589 26647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35537 every 8 connection(s)
I20260807 06:54:25.123457 26557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:25.128561 26650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:25.129899 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 26557
I20260807 06:54:25.130100 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:25.130528 27865 external_mini_cluster.cc:1428] Setting key 124ddb2bc7bc370d40c39c39cfb5a9f1
I20260807 06:54:25.140064 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:25.132489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:34921 (local address 127.0.0.1:35537)
0807 06:54:25.133260 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:25.133272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:25.133313 (+ 41us) server_negotiation.cc:407] Connection header received
0807 06:54:25.136148 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:25.136179 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:25.136636 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:25.137122 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:25.138094 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:25.138102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:25.138120 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:25.138135 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:25.138201 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:25.138233 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:25.138442 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:25.138573 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 06:54:25.138811 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":325,"threads_started":1}
I20260807 06:54:25.144058 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:25.144483 26650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c: Bootstrap starting.
I20260807 06:54:25.147729 26650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:25.148928 26650 log.cc:824] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:25.151504 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:25.148681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:57995 (local address 127.0.0.1:35537)
0807 06:54:25.148928 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:25.148937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:25.149390 (+ 453us) server_negotiation.cc:407] Connection header received
0807 06:54:25.149636 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:25.149642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:25.149716 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:25.149839 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:25.150879 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:25.150883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:25.150887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:25.150901 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:25.150938 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:25.150953 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:25.151043 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:25.151165 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 06:54:25.151211 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:54:25.152506 26650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c: No bootstrap required, opened a new log
I20260807 06:54:25.160104 26650 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0de3d7d9c5497fa6157632406aec8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260807 06:54:25.160485 26650 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:25.160588 26650 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0de3d7d9c5497fa6157632406aec8c, State: Initialized, Role: FOLLOWER
I20260807 06:54:25.161229 26650 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0de3d7d9c5497fa6157632406aec8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260807 06:54:25.161465 26650 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:25.161626 26650 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:25.161828 26650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:25.164213 26650 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0de3d7d9c5497fa6157632406aec8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260807 06:54:25.164726 26650 leader_election.cc:302] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa0de3d7d9c5497fa6157632406aec8c; no voters:
I20260807 06:54:25.165335 26650 leader_election.cc:288] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:25.165494 26665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:25.166605 26665 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [term 1 LEADER]: Becoming Leader. State: Replica: aa0de3d7d9c5497fa6157632406aec8c, State: Running, Role: LEADER
I20260807 06:54:25.167161 26665 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0de3d7d9c5497fa6157632406aec8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260807 06:54:25.167675 26650 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:25.169881 26666 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0de3d7d9c5497fa6157632406aec8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0de3d7d9c5497fa6157632406aec8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260807 06:54:25.170224 26666 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:25.170032 26667 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0de3d7d9c5497fa6157632406aec8c. Latest consensus state: current_term: 1 leader_uuid: "aa0de3d7d9c5497fa6157632406aec8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0de3d7d9c5497fa6157632406aec8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260807 06:54:25.170478 26667 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:25.171329 26673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:25.178249 26673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:25.194343 26673 catalog_manager.cc:1347] Generated new cluster ID: 387e8d2e5dbd433a9589bec90106582f
I20260807 06:54:25.194512 26673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:25.210527 26673 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:25.211125 26673 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:25.228667 26673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0de3d7d9c5497fa6157632406aec8c: Generated new TSK 0
I20260807 06:54:25.229808 26673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:25.244035 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26557
W20260807 06:54:25.245621 26642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.97:34921
W20260807 06:54:25.246361 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26557: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:25.246519 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 26557
2026-08-07T06:54:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:25 dist-test-slave-3kk6 krb5kdc[26694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:25 dist-test-slave-3kk6 krb5kdc[26694](info): set up 2 sockets
Aug 07 06:54:25 dist-test-slave-3kk6 krb5kdc[26694](info): commencing operation
krb5kdc: starting...
W20260807 06:54:26.593590 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:26 dist-test-slave-3kk6 krb5kdc[26694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:26.975008 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:44831
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:27.342741 26710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:27.343084 26710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:27.343256 26710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:27.351397 26710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:27.351534 26710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:27.351655 26710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:27.351735 26710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:27.351799 26710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:27.362221 26710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:44831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:27.363700 26710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:27.365098 26710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:27.377652 26720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:27.377571 26726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:27.377616 26718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:27.379446 26710 server_base.cc:1034] running on GCE node
I20260807 06:54:27.380336 26710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:27.381950 26710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:27.383158 26710 hybrid_clock.cc:648] HybridClock initialized: now 1786085667383100 us; error 56 us; skew 500 ppm
I20260807 06:54:27.383550 26710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:27 dist-test-slave-3kk6 krb5kdc[26694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085667, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:27.387171 26710 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:27.388803 26710 webserver.cc:460] Webserver started at http://127.27.54.126:37173/ using document root <none> and password file <none>
I20260807 06:54:27.389526 26710 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:27.389662 26710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:27.389998 26710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:27.393253 26710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "b615f94494104b3bba7f92304a934fe8"
format_stamp: "Formatted at 2026-08-07 06:54:27 on dist-test-slave-3kk6"
server_key: "c7f3c49c73f9f44083199f722ec9b216"
server_key_iv: "1fc05fd43e50f06668eb9a6c2dfaff4b"
server_key_version: "encryptionkey@0"
I20260807 06:54:27.394062 26710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "b615f94494104b3bba7f92304a934fe8"
format_stamp: "Formatted at 2026-08-07 06:54:27 on dist-test-slave-3kk6"
server_key: "c7f3c49c73f9f44083199f722ec9b216"
server_key_iv: "1fc05fd43e50f06668eb9a6c2dfaff4b"
server_key_version: "encryptionkey@0"
I20260807 06:54:27.399647 26710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 06:54:27.403331 26735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:27.404682 26710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:27.404882 26710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "b615f94494104b3bba7f92304a934fe8"
format_stamp: "Formatted at 2026-08-07 06:54:27 on dist-test-slave-3kk6"
server_key: "c7f3c49c73f9f44083199f722ec9b216"
server_key_iv: "1fc05fd43e50f06668eb9a6c2dfaff4b"
server_key_version: "encryptionkey@0"
I20260807 06:54:27.405102 26710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:27.424506 26710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:27.428889 26710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:27.429198 26710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:27.449394 26710 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:44831
I20260807 06:54:27.449419 26823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:44831 every 8 connection(s)
I20260807 06:54:27.450968 26710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:27.453032 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 26710
I20260807 06:54:27.453245 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:27.453841 27865 external_mini_cluster.cc:1428] Setting key edd9eeb659d3de6aa933b55804e3983c
I20260807 06:54:27.456189 26824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:54:27 dist-test-slave-3kk6 krb5kdc[26694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:54:27.470845 26824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8: Bootstrap starting.
I20260807 06:54:27.474915 26824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:27.476066 26824 log.cc:824] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:27.479432 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:27.455877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52149 (local address 127.27.54.126:44831)
0807 06:54:27.456720 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:27.456732 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:27.456798 (+ 66us) server_negotiation.cc:407] Connection header received
0807 06:54:27.458506 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:27.458529 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:27.459021 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:27.459382 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:27.461469 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:27.462970 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:27.464810 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:27.465517 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:27.465654 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:27.471119 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:27.471129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:27.471145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:27.471199 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:27.474517 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:27.475723 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:27.475732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:27.475739 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:27.475929 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:27.476506 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:27.476516 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:27.476518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:27.477087 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:27.477449 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:27.477948 (+ 499us) server_negotiation.cc:299] Negotiation successful
0807 06:54:27.478300 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":382,"threads_started":1}
I20260807 06:54:27.479516 26824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8: No bootstrap required, opened a new log
I20260807 06:54:27.483541 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:27.488241 26824 raft_consensus.cc:348] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615f94494104b3bba7f92304a934fe8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44831 } }
I20260807 06:54:27.488746 26824 raft_consensus.cc:374] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:27.488879 26824 raft_consensus.cc:729] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b615f94494104b3bba7f92304a934fe8, State: Initialized, Role: FOLLOWER
I20260807 06:54:27.489547 26824 consensus_queue.cc:260] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615f94494104b3bba7f92304a934fe8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44831 } }
I20260807 06:54:27.489806 26824 raft_consensus.cc:388] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:27.489974 26824 raft_consensus.cc:482] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:27.490151 26824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:27.493309 26824 raft_consensus.cc:504] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615f94494104b3bba7f92304a934fe8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44831 } }
I20260807 06:54:27.493981 26824 leader_election.cc:302] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b615f94494104b3bba7f92304a934fe8; no voters:
I20260807 06:54:27.494793 26824 leader_election.cc:288] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:27.494931 26838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:27.498821 26838 raft_consensus.cc:686] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [term 1 LEADER]: Becoming Leader. State: Replica: b615f94494104b3bba7f92304a934fe8, State: Running, Role: LEADER
I20260807 06:54:27.499759 26838 consensus_queue.cc:237] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615f94494104b3bba7f92304a934fe8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44831 } }
I20260807 06:54:27.501740 26824 sys_catalog.cc:564] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:27.503453 26839 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b615f94494104b3bba7f92304a934fe8. Latest consensus state: current_term: 1 leader_uuid: "b615f94494104b3bba7f92304a934fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615f94494104b3bba7f92304a934fe8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44831 } } }
I20260807 06:54:27.503918 26839 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:27.503428 26841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b615f94494104b3bba7f92304a934fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b615f94494104b3bba7f92304a934fe8" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 44831 } } }
I20260807 06:54:27.505502 26841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:27.509295 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:27.488086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52883 (local address 127.27.54.126:44831)
0807 06:54:27.488307 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:27.488314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:27.488766 (+ 452us) server_negotiation.cc:407] Connection header received
0807 06:54:27.488954 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:27.488961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:27.489049 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:27.489176 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:27.490971 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:27.492211 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:27.493458 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:27.494118 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:27.494273 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:27.498594 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:27.498605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:27.498613 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:27.498723 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:27.503784 (+ 5061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:27.505459 (+ 1675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:27.505468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:27.505474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:27.505744 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:27.506571 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:27.506579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:27.506584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:27.507266 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:27.507795 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:27.508370 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 06:54:27.508651 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:54:27.515321 26848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:27.522046 26848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:27.532136 26848 catalog_manager.cc:1347] Generated new cluster ID: de5ae0e7ef874a0e813c54db01ff9590
I20260807 06:54:27.532311 26848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:27.559131 26848 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:27.560732 26848 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:27.578188 26848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b615f94494104b3bba7f92304a934fe8: Generated new TSK 0
I20260807 06:54:27.579281 26848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:27.594064 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26710
I20260807 06:54:27.596036 26818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:27.694049 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 26710
W20260807 06:54:27.721522 26692 connection.cc:535] client connection to 127.27.54.126:44831 recv error: Network error: failed to read from TLS socket (remote: 127.27.54.126:44831): Connection reset by peer (error 104)
2026-08-07T06:54:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:27 dist-test-slave-3kk6 krb5kdc[26874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:27 dist-test-slave-3kk6 krb5kdc[26874](info): set up 2 sockets
Aug 07 06:54:27 dist-test-slave-3kk6 krb5kdc[26874](info): commencing operation
krb5kdc: starting...
W20260807 06:54:29.059958 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.319s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:29 dist-test-slave-3kk6 krb5kdc[26874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:29.438535 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:43001
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:29.810325 26897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:29.810712 26897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:29.810845 26897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:29.818449 26897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:29.818593 26897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:29.818642 26897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:29.818809 26897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:29.818856 26897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:29.829169 26897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:43001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:29.830549 26897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:29.832026 26897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:29.844461 26902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:29.844398 26903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:29.844367 26908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:29.845777 26897 server_base.cc:1034] running on GCE node
I20260807 06:54:29.846776 26897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:29.848336 26897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:29.849617 26897 hybrid_clock.cc:648] HybridClock initialized: now 1786085669849555 us; error 51 us; skew 500 ppm
I20260807 06:54:29.850077 26897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:29 dist-test-slave-3kk6 krb5kdc[26874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:29.853623 26897 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:29.855326 26897 webserver.cc:460] Webserver started at http://127.27.54.126:40597/ using document root <none> and password file <none>
I20260807 06:54:29.856037 26897 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:29.856148 26897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:29.856429 26897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:29.859664 26897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "9d20042972de4e56939767b8b5411851"
format_stamp: "Formatted at 2026-08-07 06:54:29 on dist-test-slave-3kk6"
server_key: "3787e8e9f5db76418d0883dbe080dbee"
server_key_iv: "d6c29e095eca2549707b8cf8220c6ada"
server_key_version: "encryptionkey@0"
I20260807 06:54:29.860438 26897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "9d20042972de4e56939767b8b5411851"
format_stamp: "Formatted at 2026-08-07 06:54:29 on dist-test-slave-3kk6"
server_key: "3787e8e9f5db76418d0883dbe080dbee"
server_key_iv: "d6c29e095eca2549707b8cf8220c6ada"
server_key_version: "encryptionkey@0"
I20260807 06:54:29.865954 26897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 06:54:29.869683 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:29.871215 26897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:54:29.871436 26897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "9d20042972de4e56939767b8b5411851"
format_stamp: "Formatted at 2026-08-07 06:54:29 on dist-test-slave-3kk6"
server_key: "3787e8e9f5db76418d0883dbe080dbee"
server_key_iv: "d6c29e095eca2549707b8cf8220c6ada"
server_key_version: "encryptionkey@0"
I20260807 06:54:29.871675 26897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:29.900306 26897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:29.905213 26897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:29.905543 26897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:29.926333 26897 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:43001
I20260807 06:54:29.926352 26998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:43001 every 8 connection(s)
I20260807 06:54:29.927896 26897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:29.931985 27000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:29.937228 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 26897
I20260807 06:54:29.937479 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:29.938041 27865 external_mini_cluster.cc:1428] Setting key 1dadc2c3dff15c6ba722a9f1caaaf1c4
I20260807 06:54:29.944059 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851: Bootstrap starting.
I20260807 06:54:29.947403 27000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:29.948585 27000 log.cc:824] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:29.951917 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851: No bootstrap required, opened a new log
I20260807 06:54:29.952713 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:29.940273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59093 (local address 127.27.54.126:43001)
0807 06:54:29.941280 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:29.941293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:29.941345 (+ 52us) server_negotiation.cc:407] Connection header received
0807 06:54:29.943260 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:29.943284 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:29.943681 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:29.944123 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:29.944758 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:29.946129 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:29.947741 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:29.948489 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:29.948723 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:29.950097 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:29.950105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:29.950121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:29.950778 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:29.950865 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:29.950892 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:29.951079 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:29.951377 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 06:54:29.951561 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":440,"threads_started":1}
I20260807 06:54:29.957365 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:29.961156 27000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d20042972de4e56939767b8b5411851" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43001 } }
I20260807 06:54:29.961637 27000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:29.961779 27000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d20042972de4e56939767b8b5411851, State: Initialized, Role: FOLLOWER
I20260807 06:54:29.962602 27000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d20042972de4e56939767b8b5411851" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43001 } }
I20260807 06:54:29.962997 27000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:29.963171 27000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:29.963384 27000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:29.965926 27000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d20042972de4e56939767b8b5411851" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43001 } }
I20260807 06:54:29.966507 27000 leader_election.cc:302] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d20042972de4e56939767b8b5411851; no voters:
I20260807 06:54:29.967329 27000 leader_election.cc:288] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:29.967492 27012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:29.969815 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:29.962346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38019 (local address 127.27.54.126:43001)
0807 06:54:29.962533 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:29.962538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:29.963153 (+ 615us) server_negotiation.cc:407] Connection header received
0807 06:54:29.963429 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:29.963435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:29.963527 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:29.963645 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:29.964224 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:29.965043 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:29.966573 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:29.967160 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:29.967338 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:29.968866 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:29.968875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:29.968883 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:29.968910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:29.968959 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:29.968978 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:29.969069 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:29.969366 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 06:54:29.969424 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 06:54:29.970368 27000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:29.974643 27012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [term 1 LEADER]: Becoming Leader. State: Replica: 9d20042972de4e56939767b8b5411851, State: Running, Role: LEADER
I20260807 06:54:29.975719 27012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d20042972de4e56939767b8b5411851" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43001 } }
I20260807 06:54:29.980036 27014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d20042972de4e56939767b8b5411851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d20042972de4e56939767b8b5411851" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43001 } } }
I20260807 06:54:29.980136 27015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d20042972de4e56939767b8b5411851. Latest consensus state: current_term: 1 leader_uuid: "9d20042972de4e56939767b8b5411851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d20042972de4e56939767b8b5411851" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 43001 } } }
I20260807 06:54:29.980554 27014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:29.980587 27015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:29.982359 27025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:29.988297 27025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:29.998895 27025 catalog_manager.cc:1347] Generated new cluster ID: fb96fa769f61431ea1a091333f6203e0
I20260807 06:54:29.999038 27025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:30.033114 27025 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:30.034551 27025 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:30.050889 27025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d20042972de4e56939767b8b5411851: Generated new TSK 0
I20260807 06:54:30.052088 27025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:30.073750 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26897
W20260807 06:54:30.075243 26993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59093
W20260807 06:54:30.076175 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 26897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:30.076356 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 26897
W20260807 06:54:30.106441 26871 connection.cc:535] client connection to 127.27.54.126:43001 recv error: Network error: failed to read from TLS socket (remote: 127.27.54.126:43001): Connection reset by peer (error 104)
2026-08-07T06:54:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:30 dist-test-slave-3kk6 krb5kdc[27041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:30 dist-test-slave-3kk6 krb5kdc[27041](info): set up 2 sockets
Aug 07 06:54:30 dist-test-slave-3kk6 krb5kdc[27041](info): commencing operation
krb5kdc: starting...
W20260807 06:54:31.431694 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.001s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:31 dist-test-slave-3kk6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:31.806592 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34485
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:32.170694 27061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:32.171028 27061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:32.171128 27061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:32.178403 27061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:32.178525 27061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:32.178574 27061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:32.178634 27061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:32.178704 27061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:32.188562 27061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:34485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:34485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:32.189850 27061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:32.191185 27061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:32.203121 27069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:32.203146 27072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:32.203995 27070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:32.204241 27061 server_base.cc:1034] running on GCE node
I20260807 06:54:32.205042 27061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:32.206436 27061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:32.207644 27061 hybrid_clock.cc:648] HybridClock initialized: now 1786085672207598 us; error 38 us; skew 500 ppm
I20260807 06:54:32.208029 27061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:32 dist-test-slave-3kk6 krb5kdc[27041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085672, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:32.211344 27061 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:32.212924 27061 webserver.cc:460] Webserver started at http://127.27.54.126:41857/ using document root <none> and password file <none>
I20260807 06:54:32.213595 27061 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:32.213753 27061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:32.214074 27061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:32.217242 27061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "c7effe98157f466cae3837c050cf05ad"
format_stamp: "Formatted at 2026-08-07 06:54:32 on dist-test-slave-3kk6"
server_key: "8e38ffe4f8cff05023c3f0fbe26c8c54"
server_key_iv: "ee457674eb7d7f2e075af46646847d36"
server_key_version: "encryptionkey@0"
I20260807 06:54:32.218056 27061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "c7effe98157f466cae3837c050cf05ad"
format_stamp: "Formatted at 2026-08-07 06:54:32 on dist-test-slave-3kk6"
server_key: "8e38ffe4f8cff05023c3f0fbe26c8c54"
server_key_iv: "ee457674eb7d7f2e075af46646847d36"
server_key_version: "encryptionkey@0"
I20260807 06:54:32.223572 27061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 06:54:32.227150 27088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:32.228531 27061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 06:54:32.228734 27061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "c7effe98157f466cae3837c050cf05ad"
format_stamp: "Formatted at 2026-08-07 06:54:32 on dist-test-slave-3kk6"
server_key: "8e38ffe4f8cff05023c3f0fbe26c8c54"
server_key_iv: "ee457674eb7d7f2e075af46646847d36"
server_key_version: "encryptionkey@0"
I20260807 06:54:32.228961 27061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:32.234999 27061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:32.239578 27061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:32.239894 27061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:32.259377 27061 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:34485
I20260807 06:54:32.259430 27159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:34485 every 8 connection(s)
I20260807 06:54:32.260862 27061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:32.263926 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27061
I20260807 06:54:32.264129 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:32.264717 27865 external_mini_cluster.cc:1428] Setting key a412d5ced2e5da7a09e9dad1c846a67e
I20260807 06:54:32.265139 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:32.273515 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:32.266678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41905 (local address 127.27.54.126:34485)
0807 06:54:32.267496 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:32.267509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:32.267549 (+ 40us) server_negotiation.cc:407] Connection header received
0807 06:54:32.269888 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:32.269910 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:32.270254 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:32.270812 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:32.271890 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:32.271896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:32.271911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:32.271932 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:32.271991 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:32.272010 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:32.272265 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:32.272425 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 06:54:32.272599 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":369,"threads_started":1}
I20260807 06:54:32.277518 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:32.279079 27160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad: Bootstrap starting.
I20260807 06:54:32.282605 27160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:32.284096 27160 log.cc:824] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:32.284930 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:32.282122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42445 (local address 127.27.54.126:34485)
0807 06:54:32.282329 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:32.282335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:32.282843 (+ 508us) server_negotiation.cc:407] Connection header received
0807 06:54:32.282992 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:32.282999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:32.283080 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:32.283216 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:32.284145 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:32.284151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:32.284155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:32.284174 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:32.284224 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:32.284245 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:32.284374 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:32.284489 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 06:54:32.284547 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 06:54:32.287555 27160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad: No bootstrap required, opened a new log
I20260807 06:54:32.295676 27160 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7effe98157f466cae3837c050cf05ad" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:54:32.296068 27160 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:32.296146 27160 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7effe98157f466cae3837c050cf05ad, State: Initialized, Role: FOLLOWER
I20260807 06:54:32.296662 27160 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7effe98157f466cae3837c050cf05ad" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:54:32.296885 27160 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:32.296993 27160 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:32.297119 27160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:32.299703 27160 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7effe98157f466cae3837c050cf05ad" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:54:32.300328 27160 leader_election.cc:302] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7effe98157f466cae3837c050cf05ad; no voters:
I20260807 06:54:32.300968 27160 leader_election.cc:288] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:32.301123 27173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:32.301843 27173 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [term 1 LEADER]: Becoming Leader. State: Replica: c7effe98157f466cae3837c050cf05ad, State: Running, Role: LEADER
I20260807 06:54:32.302456 27173 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7effe98157f466cae3837c050cf05ad" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } }
I20260807 06:54:32.303236 27160 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:32.305464 27174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7effe98157f466cae3837c050cf05ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7effe98157f466cae3837c050cf05ad" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } } }
I20260807 06:54:32.305944 27174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:32.307595 27180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:32.307456 27175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7effe98157f466cae3837c050cf05ad. Latest consensus state: current_term: 1 leader_uuid: "c7effe98157f466cae3837c050cf05ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7effe98157f466cae3837c050cf05ad" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 34485 } } }
I20260807 06:54:32.307783 27175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:32.312840 27180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:32.327176 27180 catalog_manager.cc:1347] Generated new cluster ID: bbe7c321408f44b2aeb19fc46e67eda7
I20260807 06:54:32.327316 27180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:32.344769 27180 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:32.345240 27180 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:32.353372 27180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7effe98157f466cae3837c050cf05ad: Generated new TSK 0
I20260807 06:54:32.354211 27180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:32.363729 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27061
W20260807 06:54:32.364880 27151 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41905
W20260807 06:54:32.365669 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:32.365856 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27061
2026-08-07T06:54:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:32 dist-test-slave-3kk6 krb5kdc[27202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:32 dist-test-slave-3kk6 krb5kdc[27202](info): set up 2 sockets
Aug 07 06:54:32 dist-test-slave-3kk6 krb5kdc[27202](info): commencing operation
krb5kdc: starting...
W20260807 06:54:33.712229 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:33 dist-test-slave-3kk6 krb5kdc[27202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:34.094700 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41601
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:41601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:34.458199 27219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:34.458508 27219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:34.458736 27219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:34.466058 27219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:34.466228 27219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:34.466352 27219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:34.466452 27219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:34.466516 27219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:34.476800 27219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:41601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:34.478101 27219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:34.479408 27219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:34.491931 27233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:34.492163 27236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:34.492472 27230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:34.492841 27219 server_base.cc:1034] running on GCE node
I20260807 06:54:34.493700 27219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:34.495270 27219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:34.496515 27219 hybrid_clock.cc:648] HybridClock initialized: now 1786085674496463 us; error 61 us; skew 500 ppm
I20260807 06:54:34.496891 27219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:34 dist-test-slave-3kk6 krb5kdc[27202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085674, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:34.500252 27219 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:34.501907 27219 webserver.cc:460] Webserver started at http://127.27.54.126:39359/ using document root <none> and password file <none>
I20260807 06:54:34.502732 27219 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:34.502892 27219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:34.503221 27219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:34.506376 27219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "803b7f92af2049628f3982c0caeee728"
format_stamp: "Formatted at 2026-08-07 06:54:34 on dist-test-slave-3kk6"
server_key: "9cc56b8ff09397f9ddaba08c4a8aa9db"
server_key_iv: "bd3d8c90cfe0de1a2b30615974d66223"
server_key_version: "encryptionkey@0"
I20260807 06:54:34.507243 27219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "803b7f92af2049628f3982c0caeee728"
format_stamp: "Formatted at 2026-08-07 06:54:34 on dist-test-slave-3kk6"
server_key: "9cc56b8ff09397f9ddaba08c4a8aa9db"
server_key_iv: "bd3d8c90cfe0de1a2b30615974d66223"
server_key_version: "encryptionkey@0"
I20260807 06:54:34.512646 27219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 06:54:34.516412 27251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:34.517793 27219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 06:54:34.518008 27219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "803b7f92af2049628f3982c0caeee728"
format_stamp: "Formatted at 2026-08-07 06:54:34 on dist-test-slave-3kk6"
server_key: "9cc56b8ff09397f9ddaba08c4a8aa9db"
server_key_iv: "bd3d8c90cfe0de1a2b30615974d66223"
server_key_version: "encryptionkey@0"
I20260807 06:54:34.518229 27219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:34.536087 27219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:34.540342 27219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:34.540676 27219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:34.560877 27219 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:41601
I20260807 06:54:34.560913 27321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:41601 every 8 connection(s)
I20260807 06:54:34.562373 27219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:34.564678 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27219
I20260807 06:54:34.564888 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:34.565471 27865 external_mini_cluster.cc:1428] Setting key b6ef41a5dab9bdd3f7818aa660a083f1
I20260807 06:54:34.567157 27322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 06:54:34 dist-test-slave-3kk6 krb5kdc[27202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:54:34.581722 27322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728: Bootstrap starting.
I20260807 06:54:34.585093 27322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:34.586230 27322 log.cc:824] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:34.590124 27322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728: No bootstrap required, opened a new log
I20260807 06:54:34.590643 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:34.567455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34817 (local address 127.27.54.126:41601)
0807 06:54:34.568292 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:34.568303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:34.568346 (+ 43us) server_negotiation.cc:407] Connection header received
0807 06:54:34.569961 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:34.569983 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:34.570386 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:34.570930 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:34.572704 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:34.574086 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:34.576004 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:34.576727 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:34.576874 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:34.582182 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:34.582192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:34.582209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:34.582253 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:34.585707 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:34.586522 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:34.586532 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:34.586541 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:34.586790 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:34.587368 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:34.587372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:34.587374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:34.588023 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:34.588327 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:34.588794 (+ 467us) server_negotiation.cc:299] Negotiation successful
0807 06:54:34.589274 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":399,"threads_started":1}
I20260807 06:54:34.594689 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:34.600595 27322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803b7f92af2049628f3982c0caeee728" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41601 } }
I20260807 06:54:34.601047 27322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:34.601181 27322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803b7f92af2049628f3982c0caeee728, State: Initialized, Role: FOLLOWER
I20260807 06:54:34.601843 27322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803b7f92af2049628f3982c0caeee728" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41601 } }
I20260807 06:54:34.602088 27322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:34.602246 27322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:34.602442 27322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:34.604885 27322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803b7f92af2049628f3982c0caeee728" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41601 } }
I20260807 06:54:34.605435 27322 leader_election.cc:302] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 803b7f92af2049628f3982c0caeee728; no voters:
I20260807 06:54:34.606185 27322 leader_election.cc:288] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:34.606292 27334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:34.608116 27334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [term 1 LEADER]: Becoming Leader. State: Replica: 803b7f92af2049628f3982c0caeee728, State: Running, Role: LEADER
I20260807 06:54:34.608868 27322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:34.608767 27334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803b7f92af2049628f3982c0caeee728" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41601 } }
I20260807 06:54:34.617039 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:34.600351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41761 (local address 127.27.54.126:41601)
0807 06:54:34.600545 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:34.600553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:34.601164 (+ 611us) server_negotiation.cc:407] Connection header received
0807 06:54:34.601404 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:34.601414 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:34.601483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:34.601601 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:34.603455 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:34.604245 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:34.605691 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:34.606201 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:34.606362 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:34.609490 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:34.609500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:34.609504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:34.609537 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:34.613030 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:34.613846 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:34.613853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:34.613857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:34.614005 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:34.614445 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:34.614451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:34.614455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:34.614914 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:34.615775 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:34.616096 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 06:54:34.616369 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:54:34.617645 27335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "803b7f92af2049628f3982c0caeee728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803b7f92af2049628f3982c0caeee728" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41601 } } }
I20260807 06:54:34.617630 27336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 803b7f92af2049628f3982c0caeee728. Latest consensus state: current_term: 1 leader_uuid: "803b7f92af2049628f3982c0caeee728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803b7f92af2049628f3982c0caeee728" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 41601 } } }
I20260807 06:54:34.618135 27335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:34.618135 27336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728 [sys.catalog]: This master's current role is: LEADER
W20260807 06:54:34.625716 27352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:54:34.625887 27352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:54:34.626039 27347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:34.627810 27347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:34.636508 27347 catalog_manager.cc:1347] Generated new cluster ID: 0502d08db38147b3a808797ac0479135
I20260807 06:54:34.636659 27347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:34.663982 27347 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:34.665351 27347 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:34.678309 27347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 803b7f92af2049628f3982c0caeee728: Generated new TSK 0
I20260807 06:54:34.679311 27347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:34.688417 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27219
I20260807 06:54:34.690357 27316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:34.784098 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27219
2026-08-07T06:54:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:34 dist-test-slave-3kk6 krb5kdc[27362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:34 dist-test-slave-3kk6 krb5kdc[27362](info): set up 2 sockets
Aug 07 06:54:34 dist-test-slave-3kk6 krb5kdc[27362](info): commencing operation
krb5kdc: starting...
W20260807 06:54:36.162936 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:36 dist-test-slave-3kk6 krb5kdc[27362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:36.537814 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39751
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:36.903945 27380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:36.904301 27380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:36.904402 27380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:36.911909 27380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:36.912051 27380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:36.912103 27380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:36.912181 27380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:36.912230 27380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:36.922580 27380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:36.924119 27380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:36.925495 27380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:36.938172 27390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:36.938355 27392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:36.938210 27389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:36.938941 27380 server_base.cc:1034] running on GCE node
I20260807 06:54:36.939981 27380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:36.941536 27380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:36.942764 27380 hybrid_clock.cc:648] HybridClock initialized: now 1786085676942722 us; error 72 us; skew 500 ppm
I20260807 06:54:36.943145 27380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:36 dist-test-slave-3kk6 krb5kdc[27362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:36.951012 27380 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:36.952615 27380 webserver.cc:460] Webserver started at http://127.27.54.126:34727/ using document root <none> and password file <none>
I20260807 06:54:36.953305 27380 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:36.953414 27380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:36.953689 27380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:36.957055 27380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "8ceaec723ef24c799fce9240b28dbbe7"
format_stamp: "Formatted at 2026-08-07 06:54:36 on dist-test-slave-3kk6"
server_key: "7e5a9a000c7ceae75648758d7192f73c"
server_key_iv: "334ee64765082f22a776411ebc39942b"
server_key_version: "encryptionkey@0"
I20260807 06:54:36.957828 27380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "8ceaec723ef24c799fce9240b28dbbe7"
format_stamp: "Formatted at 2026-08-07 06:54:36 on dist-test-slave-3kk6"
server_key: "7e5a9a000c7ceae75648758d7192f73c"
server_key_iv: "334ee64765082f22a776411ebc39942b"
server_key_version: "encryptionkey@0"
I20260807 06:54:36.963354 27380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 06:54:36.967073 27401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:36.968513 27380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:36.968724 27380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "8ceaec723ef24c799fce9240b28dbbe7"
format_stamp: "Formatted at 2026-08-07 06:54:36 on dist-test-slave-3kk6"
server_key: "7e5a9a000c7ceae75648758d7192f73c"
server_key_iv: "334ee64765082f22a776411ebc39942b"
server_key_version: "encryptionkey@0"
I20260807 06:54:36.968946 27380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:36.995858 27380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:37.000201 27380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:37.000537 27380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:37.020293 27380 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:39751
I20260807 06:54:37.020303 27476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:39751 every 8 connection(s)
I20260807 06:54:37.021777 27380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:37.025640 27477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:37.026705 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27380
I20260807 06:54:37.026906 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:37.027382 27865 external_mini_cluster.cc:1428] Setting key 5470b02a2656c0cd7c625fa75bb8dd16
I20260807 06:54:37.039866 27477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7: Bootstrap starting.
I20260807 06:54:37.041726 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:37.029311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59577 (local address 127.27.54.126:39751)
0807 06:54:37.030255 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:37.030274 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:37.030331 (+ 57us) server_negotiation.cc:407] Connection header received
0807 06:54:37.032091 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:37.032116 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:37.032535 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:37.032893 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:37.033517 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:37.034960 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:37.036923 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:37.037663 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:37.037800 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:37.039268 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:37.039276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:37.039290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:37.039888 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:37.039983 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:37.040044 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:37.040170 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:37.040409 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 06:54:37.040736 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":400,"threads_started":1}
I20260807 06:54:37.043622 27477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:37.044835 27477 log.cc:824] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:37.045730 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:37.047794 27477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7: No bootstrap required, opened a new log
I20260807 06:54:37.057257 27481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:37.050244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41585 (local address 127.27.54.126:39751)
0807 06:54:37.050443 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:37.050450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:37.050989 (+ 539us) server_negotiation.cc:407] Connection header received
0807 06:54:37.051141 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:37.051149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:37.051239 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:37.051358 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:37.051919 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:37.053030 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:37.054407 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:37.055140 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:37.055283 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:37.056434 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:37.056440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:37.056445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:37.056468 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:37.056517 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:37.056538 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:37.056629 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:37.056823 (+ 194us) server_negotiation.cc:299] Negotiation successful
0807 06:54:37.056918 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 06:54:37.057448 27477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceaec723ef24c799fce9240b28dbbe7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39751 } }
I20260807 06:54:37.057899 27477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:37.058032 27477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ceaec723ef24c799fce9240b28dbbe7, State: Initialized, Role: FOLLOWER
I20260807 06:54:37.058616 27477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceaec723ef24c799fce9240b28dbbe7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39751 } }
I20260807 06:54:37.058941 27477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:37.059098 27477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:37.059276 27477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:37.061542 27477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceaec723ef24c799fce9240b28dbbe7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39751 } }
I20260807 06:54:37.062055 27477 leader_election.cc:302] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ceaec723ef24c799fce9240b28dbbe7; no voters:
I20260807 06:54:37.062755 27477 leader_election.cc:288] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:37.062924 27497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:37.064957 27477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:37.065073 27497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [term 1 LEADER]: Becoming Leader. State: Replica: 8ceaec723ef24c799fce9240b28dbbe7, State: Running, Role: LEADER
I20260807 06:54:37.065686 27497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceaec723ef24c799fce9240b28dbbe7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39751 } }
I20260807 06:54:37.068957 27499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ceaec723ef24c799fce9240b28dbbe7. Latest consensus state: current_term: 1 leader_uuid: "8ceaec723ef24c799fce9240b28dbbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceaec723ef24c799fce9240b28dbbe7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39751 } } }
I20260807 06:54:37.069157 27498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ceaec723ef24c799fce9240b28dbbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ceaec723ef24c799fce9240b28dbbe7" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 39751 } } }
I20260807 06:54:37.069453 27499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:37.069491 27498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:37.070909 27509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:37.076704 27509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:37.085686 27509 catalog_manager.cc:1347] Generated new cluster ID: adc9f600a5dd478c9d506a9db9311a4a
I20260807 06:54:37.085830 27509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:37.121127 27509 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:37.122643 27509 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:37.133723 27509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ceaec723ef24c799fce9240b28dbbe7: Generated new TSK 0
I20260807 06:54:37.134624 27509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:37.141005 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27380
W20260807 06:54:37.142177 27470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59577
W20260807 06:54:37.142973 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:37.143121 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27380
2026-08-07T06:54:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:37 dist-test-slave-3kk6 krb5kdc[27525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:37 dist-test-slave-3kk6 krb5kdc[27525](info): set up 2 sockets
Aug 07 06:54:37 dist-test-slave-3kk6 krb5kdc[27525](info): commencing operation
krb5kdc: starting...
W20260807 06:54:38.481711 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:38 dist-test-slave-3kk6 krb5kdc[27525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:38.883276 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:42351
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:39.263401 27541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:39.263746 27541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:39.263845 27541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:39.271359 27541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:39.271499 27541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:39.271549 27541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:39.271621 27541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:39.271687 27541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:39.282014 27541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:42351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:39.283461 27541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:39.284832 27541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:39.297909 27548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:39.297958 27549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:39.297883 27551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:39.298859 27541 server_base.cc:1034] running on GCE node
I20260807 06:54:39.300021 27541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:39.301975 27541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:39.303200 27541 hybrid_clock.cc:648] HybridClock initialized: now 1786085679303161 us; error 47 us; skew 500 ppm
I20260807 06:54:39.303572 27541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:39 dist-test-slave-3kk6 krb5kdc[27525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:39.307137 27541 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:39.308809 27541 webserver.cc:460] Webserver started at http://127.27.54.126:44727/ using document root <none> and password file <none>
I20260807 06:54:39.309545 27541 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:39.309664 27541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:39.309952 27541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:39.313592 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "fa78a912975e42fca5a4d4b3293ad1a1"
format_stamp: "Formatted at 2026-08-07 06:54:39 on dist-test-slave-3kk6"
server_key: "c3eca732687f685bb65aebdab91fd7ad"
server_key_iv: "302878350fbf3b891d476684d25c2034"
server_key_version: "encryptionkey@0"
I20260807 06:54:39.314917 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "fa78a912975e42fca5a4d4b3293ad1a1"
format_stamp: "Formatted at 2026-08-07 06:54:39 on dist-test-slave-3kk6"
server_key: "c3eca732687f685bb65aebdab91fd7ad"
server_key_iv: "302878350fbf3b891d476684d25c2034"
server_key_version: "encryptionkey@0"
I20260807 06:54:39.321372 27541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 06:54:39.325485 27560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:39.327127 27541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 06:54:39.327361 27541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "fa78a912975e42fca5a4d4b3293ad1a1"
format_stamp: "Formatted at 2026-08-07 06:54:39 on dist-test-slave-3kk6"
server_key: "c3eca732687f685bb65aebdab91fd7ad"
server_key_iv: "302878350fbf3b891d476684d25c2034"
server_key_version: "encryptionkey@0"
I20260807 06:54:39.327607 27541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:39.334225 27541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:39.339293 27541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:39.339648 27541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:39.360339 27541 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:42351
I20260807 06:54:39.360378 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:42351 every 8 connection(s)
I20260807 06:54:39.361954 27541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:39.362524 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27541
I20260807 06:54:39.362756 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:39.363445 27865 external_mini_cluster.cc:1428] Setting key e9c68d18425542719c70c1f09335fd87
I20260807 06:54:39.366812 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:39.371970 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:39.365448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44201 (local address 127.27.54.126:42351)
0807 06:54:39.366292 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:39.366304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:39.366355 (+ 51us) server_negotiation.cc:407] Connection header received
0807 06:54:39.368200 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:39.368223 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:39.368572 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:39.368978 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:39.370186 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:39.370192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:39.370206 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:39.370221 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:39.370305 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:39.370331 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:39.370607 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:39.370776 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 06:54:39.370967 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":369,"threads_started":1}
I20260807 06:54:39.376330 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:39.379905 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1: Bootstrap starting.
I20260807 06:54:39.384086 27636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:39.384303 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:39.381294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56121 (local address 127.27.54.126:42351)
0807 06:54:39.381536 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:39.381543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:39.381965 (+ 422us) server_negotiation.cc:407] Connection header received
0807 06:54:39.382197 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:39.382202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:39.382256 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:39.382391 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:39.383511 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:39.383523 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:39.383526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 06:54:39.383543 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:39.383586 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 06:54:39.383601 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:39.383732 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:39.383878 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 06:54:39.383950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 06:54:39.385706 27636 log.cc:824] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:39.389276 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1: No bootstrap required, opened a new log
I20260807 06:54:39.397594 27636 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42351 } }
I20260807 06:54:39.398020 27636 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 06:54:39.398097 27636 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa78a912975e42fca5a4d4b3293ad1a1, State: Initialized, Role: FOLLOWER
I20260807 06:54:39.398713 27636 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42351 } }
I20260807 06:54:39.398947 27636 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 06:54:39.399086 27636 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 06:54:39.399262 27636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:39.402100 27636 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42351 } }
I20260807 06:54:39.402606 27636 leader_election.cc:302] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa78a912975e42fca5a4d4b3293ad1a1; no voters:
I20260807 06:54:39.403321 27636 leader_election.cc:288] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 06:54:39.403491 27651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:39.404224 27651 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [term 1 LEADER]: Becoming Leader. State: Replica: fa78a912975e42fca5a4d4b3293ad1a1, State: Running, Role: LEADER
I20260807 06:54:39.404877 27651 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42351 } }
I20260807 06:54:39.405579 27636 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:39.409085 27653 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa78a912975e42fca5a4d4b3293ad1a1. Latest consensus state: current_term: 1 leader_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42351 } } }
I20260807 06:54:39.409273 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa78a912975e42fca5a4d4b3293ad1a1" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 42351 } } }
I20260807 06:54:39.409633 27653 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:39.409723 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:39.410616 27662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:39.416657 27662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:39.430105 27662 catalog_manager.cc:1347] Generated new cluster ID: 679931a756ab42779ec91acaf1b4166c
I20260807 06:54:39.430274 27662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:39.465551 27662 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:39.466059 27662 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:39.472468 27662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa78a912975e42fca5a4d4b3293ad1a1: Generated new TSK 0
I20260807 06:54:39.473335 27662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 06:54:39.480597 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27541
W20260807 06:54:39.482264 27629 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44201
W20260807 06:54:39.483067 27865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 06:54:39.483245 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27541
2026-08-07T06:54:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2344 ms)
[----------] 18 tests from AuthTokenIssuingTest (42642 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:39 dist-test-slave-3kk6 krb5kdc[27676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:39 dist-test-slave-3kk6 krb5kdc[27676](info): set up 2 sockets
Aug 07 06:54:39 dist-test-slave-3kk6 krb5kdc[27676](info): commencing operation
krb5kdc: starting...
W20260807 06:54:40.837595 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.004s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:41 dist-test-slave-3kk6 krb5kdc[27676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:41.242599 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38551
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:38551,127.27.54.125:40453,127.27.54.124:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:41.616617 27693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:54:41.617063 27693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:41.617197 27693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:41.617342 27693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:41.624711 27693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:41.624866 27693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:41.624981 27693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:41.625073 27693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:41.625180 27693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:41.635401 27693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:38551,127.27.54.125:40453,127.27.54.124:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:41.636883 27693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:41.638315 27693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:41.650732 27698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:41.650826 27699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:41.650914 27701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:41.652189 27693 server_base.cc:1034] running on GCE node
I20260807 06:54:41.653074 27693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:41.654588 27693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:41.655893 27693 hybrid_clock.cc:648] HybridClock initialized: now 1786085681655833 us; error 33 us; skew 500 ppm
I20260807 06:54:41.656262 27693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:41 dist-test-slave-3kk6 krb5kdc[27676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:41.659682 27693 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:41.661383 27693 webserver.cc:460] Webserver started at http://127.27.54.126:40019/ using document root <none> and password file <none>
I20260807 06:54:41.662146 27693 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:41.662293 27693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:41.662699 27693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:41.665947 27693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "49a591bc2a4a4da28949d7443a900392"
format_stamp: "Formatted at 2026-08-07 06:54:41 on dist-test-slave-3kk6"
server_key: "74bcc7fc9f7ae7aa69a57ee97b4f6c9a"
server_key_iv: "931be310a6606e16cc0134562354a643"
server_key_version: "encryptionkey@0"
I20260807 06:54:41.666909 27693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "49a591bc2a4a4da28949d7443a900392"
format_stamp: "Formatted at 2026-08-07 06:54:41 on dist-test-slave-3kk6"
server_key: "74bcc7fc9f7ae7aa69a57ee97b4f6c9a"
server_key_iv: "931be310a6606e16cc0134562354a643"
server_key_version: "encryptionkey@0"
I20260807 06:54:41.672411 27693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 06:54:41.676111 27709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:41.677546 27693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:41.677775 27693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "49a591bc2a4a4da28949d7443a900392"
format_stamp: "Formatted at 2026-08-07 06:54:41 on dist-test-slave-3kk6"
server_key: "74bcc7fc9f7ae7aa69a57ee97b4f6c9a"
server_key_iv: "931be310a6606e16cc0134562354a643"
server_key_version: "encryptionkey@0"
I20260807 06:54:41.678027 27693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:41.699887 27693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:41.704547 27693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:41.704885 27693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:41.724865 27693 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:38551
I20260807 06:54:41.724884 27790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:38551 every 8 connection(s)
I20260807 06:54:41.726356 27693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:41.730240 27791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:41.732808 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27693
I20260807 06:54:41.733080 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:41.733783 27865 external_mini_cluster.cc:1428] Setting key 5e96edd6b550cd80438f54c3516546b0
I20260807 06:54:41.737323 27791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:41 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:54:41.761240 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.740750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.27.54.126:38551)
0807 06:54:41.742604 (+ 1854us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:41.742612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:41.742714 (+ 102us) server_negotiation.cc:407] Connection header received
0807 06:54:41.744332 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:41.744338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:41.744579 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:41.744815 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:41.746207 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:41.747526 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:41.748942 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:41.749520 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:41.749750 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:41.753355 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:41.753365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:41.753377 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:41.753413 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:41.756607 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:41.757361 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:41.757376 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:41.757378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:41.757530 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:41.757932 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:41.757940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:41.757943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:41.758455 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:41.758703 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:41.759249 (+ 546us) server_negotiation.cc:299] Negotiation successful
0807 06:54:41.759702 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":1630,"thread_start_us":1426,"threads_started":1}
I20260807 06:54:41.761260 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.740743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38551 (local address 127.0.0.1:52930)
0807 06:54:41.741649 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.741714 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:41.743924 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:41.744831 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:41.744838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:41.745616 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:41.745969 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:41.745987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:41.747694 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:41.747701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:41.748788 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:41.748801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:41.749780 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:41.749787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:41.749956 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:41.750764 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:41.750789 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:41.753084 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:41.756776 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:41.756791 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:41.756805 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:41.757225 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:41.757645 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:41.757652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:41.757655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:41.757817 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:41.758606 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:41.758614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:41.758858 (+ 244us) client_negotiation.cc:769] Sending connection context
0807 06:54:41.759126 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 06:54:41.759814 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":199,"thread_start_us":415,"threads_started":1}
I20260807 06:54:41.764696 27791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:54:41.765992 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.765581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38924)
0807 06:54:41.765762 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.765872 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 06:54:41.766304 27714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
I20260807 06:54:41.768066 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.767690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38940)
0807 06:54:41.767855 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.767922 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 06:54:41.768260 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:40453: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.54.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab.
Entry for principal kudu/127.27.54.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab.
I20260807 06:54:41.816484 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:40453
--webserver_interface=127.27.54.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab
--principal=kudu/127.27.54.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:38551,127.27.54.125:40453,127.27.54.124:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:54:41.817644 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } attempt: 1
I20260807 06:54:41.818959 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.818394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38948)
0807 06:54:41.818614 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.818817 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 06:54:41.820044 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.819647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38952)
0807 06:54:41.819804 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.819882 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:54:41.820246 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:40453: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
I20260807 06:54:41.888512 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } attempt: 2
I20260807 06:54:41.889770 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.889292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38960)
0807 06:54:41.889511 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.889619 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 06:54:41.890908 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.890468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38976)
0807 06:54:41.890613 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.890773 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 06:54:41.891168 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:40453: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
I20260807 06:54:41.982515 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } attempt: 3
I20260807 06:54:41.983904 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.983407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38982)
0807 06:54:41.983647 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.983758 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 06:54:41.985111 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:41.984719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:38998)
0807 06:54:41.984869 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:41.984942 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:54:41.985381 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:40453: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
I20260807 06:54:42.128645 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } attempt: 4
I20260807 06:54:42.130003 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.129433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:39004)
0807 06:54:42.129765 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.129862 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260807 06:54:42.131134 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.130803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:39010)
0807 06:54:42.130933 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.131014 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 06:54:42.131394 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:40453: Network error: Client connection negotiation failed: client connection to 127.27.54.125:40453: connect: Connection refused (error 111)
W20260807 06:54:42.187518 27799 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:54:42.187920 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:42.188047 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:42.188190 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:42.195652 27799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:42.195752 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:42.195847 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:42.195927 27799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:42.196038 27799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:42.206259 27799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:38551,127.27.54.125:40453,127.27.54.124:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab
--principal=kudu/127.27.54.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.54.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:42.207687 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:42.209087 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:42.221561 27814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:42.221694 27813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:42.221792 27816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:42.222520 27799 server_base.cc:1034] running on GCE node
I20260807 06:54:42.223345 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:42.224846 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:42.226120 27799 hybrid_clock.cc:648] HybridClock initialized: now 1786085682226054 us; error 45 us; skew 500 ppm
I20260807 06:54:42.226483 27799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:42.234881 27799 init.cc:377] Logged in from keytab as kudu/127.27.54.125@KRBTEST.COM (short username kudu)
I20260807 06:54:42.236608 27799 webserver.cc:460] Webserver started at http://127.27.54.125:34521/ using document root <none> and password file <none>
I20260807 06:54:42.237387 27799 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:42.237530 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:42.237898 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:42.241174 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data/instance:
uuid: "9bb5346326ef4411a0be9a0dcabc0de6"
format_stamp: "Formatted at 2026-08-07 06:54:42 on dist-test-slave-3kk6"
server_key: "cee49657cfe54612cd8e85af69e5449f"
server_key_iv: "dc5dd159f6f1e8d695848057a9e716d7"
server_key_version: "encryptionkey@0"
I20260807 06:54:42.242028 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal/instance:
uuid: "9bb5346326ef4411a0be9a0dcabc0de6"
format_stamp: "Formatted at 2026-08-07 06:54:42 on dist-test-slave-3kk6"
server_key: "cee49657cfe54612cd8e85af69e5449f"
server_key_iv: "dc5dd159f6f1e8d695848057a9e716d7"
server_key_version: "encryptionkey@0"
I20260807 06:54:42.248183 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 06:54:42.251892 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:42.253214 27799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:42.253412 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal
uuid: "9bb5346326ef4411a0be9a0dcabc0de6"
format_stamp: "Formatted at 2026-08-07 06:54:42 on dist-test-slave-3kk6"
server_key: "cee49657cfe54612cd8e85af69e5449f"
server_key_iv: "dc5dd159f6f1e8d695848057a9e716d7"
server_key_version: "encryptionkey@0"
I20260807 06:54:42.253664 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:42.277376 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:42.282285 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:42.282588 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:42.303434 27799 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.125:40453
I20260807 06:54:42.303475 27911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.125:40453 every 8 connection(s)
I20260807 06:54:42.305078 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
I20260807 06:54:42.306447 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27799
I20260807 06:54:42.306696 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-1/wal/instance
I20260807 06:54:42.307173 27865 external_mini_cluster.cc:1428] Setting key e4cebc7de5cf6c38e7a4af8543cf6eb5
I20260807 06:54:42.309432 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:42.316514 27912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:54:42.342319 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.319502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.27.54.126:38551)
0807 06:54:42.320138 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.320147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.321238 (+ 1091us) server_negotiation.cc:407] Connection header received
0807 06:54:42.323513 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.323523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.323631 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.323789 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.325976 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.327235 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.329993 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.330426 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.330545 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.335625 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.335632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.335635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.335686 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.338821 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.339724 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.339730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.339735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.339875 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.340354 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.340360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.340364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.340692 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.340867 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.341656 (+ 789us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.341908 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":336,"threads_started":1}
I20260807 06:54:42.343591 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.320055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38551 (local address 127.0.0.1:52934)
0807 06:54:42.321006 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.321062 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:42.323153 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:42.324037 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:42.324046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:42.324632 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:42.325601 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.325633 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.327481 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.327490 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.329796 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.329817 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.330744 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.330763 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.331188 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.332063 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:42.332101 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:42.335314 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:42.339008 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.339019 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.339047 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.339556 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.340006 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.340011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.340015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.340225 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.340845 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:42.340853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:42.341254 (+ 401us) client_negotiation.cc:769] Sending connection context
0807 06:54:42.341565 (+ 311us) client_negotiation.cc:240] Negotiation successful
0807 06:54:42.342148 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":417,"threads_started":1}
I20260807 06:54:42.345645 27912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.54.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.124@KRBTEST.COM" created.
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:42.364085 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.346566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:39018)
0807 06:54:42.346834 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.346854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:42.346975 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:42.348177 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:42.348181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:42.348563 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:42.348711 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.348719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.350454 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.350461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.351988 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.352010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.353056 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.353070 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.353319 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.354084 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:42.354113 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:42.356837 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:42.361505 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.361509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.361512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.361912 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.362377 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.362380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.362382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.362524 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.363348 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:42.363351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:42.363471 (+ 120us) client_negotiation.cc:769] Sending connection context
0807 06:54:42.363585 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 06:54:42.363811 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260807 06:54:42.364616 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.346900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.27.54.125:40453)
0807 06:54:42.347698 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.347704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.347730 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:54:42.347786 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.347792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.347973 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.348168 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.348877 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.350281 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.352192 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.352870 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.352994 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.357081 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.357102 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.357125 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.357175 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.361339 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.362077 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.362089 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.362095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.362261 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.362645 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.362676 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.362681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.363198 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.363443 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.363808 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.364103 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":492,"threads_started":1}
I20260807 06:54:42.367116 27912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:54:42.368518 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.368050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44464)
0807 06:54:42.368253 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.368370 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260807 06:54:42.368854 27827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.370580 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.370232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44478)
0807 06:54:42.370391 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.370474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 06:54:42.370832 27912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab.
Entry for principal kudu/127.27.54.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab.
I20260807 06:54:42.394465 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:33059
--webserver_interface=127.27.54.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab
--principal=kudu/127.27.54.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:38551,127.27.54.125:40453,127.27.54.124:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:54:42.420187 27912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 1
I20260807 06:54:42.421543 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.421026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44490)
0807 06:54:42.421252 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.421378 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 06:54:42.422699 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.422281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44502)
0807 06:54:42.422456 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.422534 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 06:54:42.422956 27912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.430724 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } attempt: 5
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:42.445830 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.431577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:39022)
0807 06:54:42.431816 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.431844 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:42.431993 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:42.432419 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:42.432427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:42.432880 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:42.433076 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.433085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.434128 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.434134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.435138 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.435152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.435929 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.435935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.436191 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.436950 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:42.436969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:42.439190 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:42.442983 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.442989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.442993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.443521 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.443961 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.443966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.443970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.444121 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.444697 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:42.444702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:42.444901 (+ 199us) client_negotiation.cc:769] Sending connection context
0807 06:54:42.445130 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 06:54:42.445462 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 06:54:42.445833 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.431718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.27.54.125:40453)
0807 06:54:42.431959 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.431964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.431989 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:54:42.432162 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.432167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.432260 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.432373 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.433242 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.433972 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.435310 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.435756 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.435900 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.439434 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.439439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.439442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.439478 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.442813 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.443681 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.443687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.443690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.443837 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.444241 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.444246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.444249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.444568 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.444734 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.445239 (+ 505us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.445470 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260807 06:54:42.447180 27791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:54:42.448554 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.448076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44516)
0807 06:54:42.448299 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.448395 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 06:54:42.449621 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.449275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44530)
0807 06:54:42.449422 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.449518 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 06:54:42.449843 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.491200 27912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 2
I20260807 06:54:42.492736 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.492002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44542)
0807 06:54:42.492307 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.492462 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260807 06:54:42.493988 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.493518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44546)
0807 06:54:42.493715 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.493817 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 06:54:42.494266 27912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.501092 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 1
I20260807 06:54:42.502326 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.501810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44560)
0807 06:54:42.502040 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.502145 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 06:54:42.503404 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.503053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44562)
0807 06:54:42.503208 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.503294 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:54:42.503634 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.571863 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 2
I20260807 06:54:42.573225 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.572619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44566)
0807 06:54:42.572885 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.573004 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 06:54:42.574303 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.573944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44572)
0807 06:54:42.574095 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.574188 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 06:54:42.574518 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.585553 27912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 3
I20260807 06:54:42.587077 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.586398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44586)
0807 06:54:42.586728 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.586857 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 06:54:42.588533 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.587965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44600)
0807 06:54:42.588254 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.588374 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260807 06:54:42.588807 27912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.680742 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 3
I20260807 06:54:42.682169 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.681510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44616)
0807 06:54:42.681808 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.681942 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260807 06:54:42.683601 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.683030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44632)
0807 06:54:42.683317 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.683434 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260807 06:54:42.683895 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.732064 27912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 4
I20260807 06:54:42.733475 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.732821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44634)
0807 06:54:42.733106 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.733265 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 06:54:42.734774 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.734339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44640)
0807 06:54:42.734503 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.734599 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 06:54:42.735054 27912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
W20260807 06:54:42.770532 27922 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:54:42.770961 27922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:42.771045 27922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:42.771124 27922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:42.778498 27922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:42.778611 27922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:42.778746 27922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:42.778802 27922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:42.778873 27922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:42.789232 27922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:38551,127.27.54.125:40453,127.27.54.124:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab
--principal=kudu/127.27.54.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.54.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:42.790596 27922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:42.792073 27922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:42.804782 27933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:42.804807 27931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:42.804857 27930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:42.805827 27922 server_base.cc:1034] running on GCE node
I20260807 06:54:42.806774 27922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:42.808266 27922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:42.809530 27922 hybrid_clock.cc:648] HybridClock initialized: now 1786085682809466 us; error 42 us; skew 500 ppm
I20260807 06:54:42.809887 27922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:42.813376 27922 init.cc:377] Logged in from keytab as kudu/127.27.54.124@KRBTEST.COM (short username kudu)
I20260807 06:54:42.815064 27922 webserver.cc:460] Webserver started at http://127.27.54.124:33033/ using document root <none> and password file <none>
I20260807 06:54:42.815832 27922 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:42.815979 27922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:42.816331 27922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:42.819622 27922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data/instance:
uuid: "468a6a7a3fa346c08210dbc49d97f7d3"
format_stamp: "Formatted at 2026-08-07 06:54:42 on dist-test-slave-3kk6"
server_key: "4d2d1049b65a0019a45575fef3f232fc"
server_key_iv: "5eb106a1bf038abf035cb65b82887f95"
server_key_version: "encryptionkey@0"
I20260807 06:54:42.820516 27922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal/instance:
uuid: "468a6a7a3fa346c08210dbc49d97f7d3"
format_stamp: "Formatted at 2026-08-07 06:54:42 on dist-test-slave-3kk6"
server_key: "4d2d1049b65a0019a45575fef3f232fc"
server_key_iv: "5eb106a1bf038abf035cb65b82887f95"
server_key_version: "encryptionkey@0"
I20260807 06:54:42.826150 27922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 06:54:42.829864 27943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:42.831382 27922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 06:54:42.831583 27922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal
uuid: "468a6a7a3fa346c08210dbc49d97f7d3"
format_stamp: "Formatted at 2026-08-07 06:54:42 on dist-test-slave-3kk6"
server_key: "4d2d1049b65a0019a45575fef3f232fc"
server_key_iv: "5eb106a1bf038abf035cb65b82887f95"
server_key_version: "encryptionkey@0"
I20260807 06:54:42.831875 27922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:42.852141 27922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:42.856999 27922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:42.857331 27922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:42.861159 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 4
I20260807 06:54:42.862429 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.861856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44656)
0807 06:54:42.862120 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.862250 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 06:54:42.863592 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.863174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44658)
0807 06:54:42.863309 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.863405 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 06:54:42.863914 27791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:33059: Network error: Client connection negotiation failed: client connection to 127.27.54.124:33059: connect: Connection refused (error 111)
I20260807 06:54:42.878232 27922 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.124:33059
I20260807 06:54:42.878258 28017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.124:33059 every 8 connection(s)
I20260807 06:54:42.879817 27922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
I20260807 06:54:42.883955 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 27922
I20260807 06:54:42.884161 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786085287136147-27865-0/minicluster-data/master-2/wal/instance
I20260807 06:54:42.884282 28018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:42.884778 27865 external_mini_cluster.cc:1428] Setting key 67073a639c702a338e7f5fd4d9d818d6
I20260807 06:54:42.894631 28018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:54:42.911245 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.886873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.27.54.124:33059)
0807 06:54:42.887779 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.887793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.887838 (+ 45us) server_negotiation.cc:407] Connection header received
0807 06:54:42.889563 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.889587 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.889990 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.890360 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.892495 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.893829 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.895748 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.896500 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.896641 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.901780 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.901793 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.901809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.901867 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.906072 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.907045 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.907058 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.907070 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.907291 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.907774 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.907786 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.907788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.908346 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.908673 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.909207 (+ 534us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.909689 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":423,"threads_started":1}
I20260807 06:54:42.915726 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.897992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:38551 (local address 127.0.0.1:52944)
0807 06:54:42.899037 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.899090 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:42.899276 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:42.899740 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:42.899745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:42.900396 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:42.900699 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.900714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.901841 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.901845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.903229 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.903242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.904299 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.904304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.904428 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.905207 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:42.905241 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:42.908063 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:42.911860 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.911869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.911875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.912307 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.913645 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.913649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.913654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.913844 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.914516 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:42.914526 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:42.914840 (+ 314us) client_negotiation.cc:769] Sending connection context
0807 06:54:42.914995 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:54:42.915251 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":669,"threads_started":1}
I20260807 06:54:42.915848 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.898039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.27.54.126:38551)
0807 06:54:42.898683 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.898702 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.899200 (+ 498us) server_negotiation.cc:407] Connection header received
0807 06:54:42.899434 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.899442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.899535 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.899664 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.900880 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.901646 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.903394 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.904082 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.904226 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.908274 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.908279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.908282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.908317 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.911725 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.912452 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.912458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.912461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.913518 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.913972 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.913975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.913977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.914307 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.914495 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.915170 (+ 675us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.915438 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":298,"threads_started":1}
I20260807 06:54:42.917500 28018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:42.932032 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.914378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.27.54.126:38551)
0807 06:54:42.915048 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.915054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.915077 (+ 23us) server_negotiation.cc:407] Connection header received
0807 06:54:42.915120 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.915126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.915210 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.915320 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.917039 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.918127 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.919541 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.919979 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.920119 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.925308 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.925313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.925316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.925350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.929002 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.929726 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.929731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.929735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.929908 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.930330 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.930335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.930338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.930771 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.931045 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.931355 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.931591 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260807 06:54:42.932526 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.918411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.27.54.125:40453)
0807 06:54:42.918617 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.918626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.918868 (+ 242us) server_negotiation.cc:407] Connection header received
0807 06:54:42.919060 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.919065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.919125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.919231 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.919970 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.920691 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.922563 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.923045 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.923144 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.926465 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.926475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.926482 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.926526 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.929784 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.930400 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.930407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.930413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.930584 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.931051 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.931056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.931060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.931335 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.931504 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.932010 (+ 506us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.932261 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:54:42.932559 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.918434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:40453 (local address 127.0.0.1:39028)
0807 06:54:42.918778 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.918794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:42.918924 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:42.919268 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:42.919273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:42.919666 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:42.919822 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.919831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.920893 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.920897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.922404 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.922424 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.923207 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.923213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.923438 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.924045 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:42.924062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:42.926264 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:42.929928 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.929932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.929934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.930302 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.930777 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.930781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.930783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.930929 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.931484 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:42.931489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:42.931658 (+ 169us) client_negotiation.cc:769] Sending connection context
0807 06:54:42.931895 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 06:54:42.932192 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260807 06:54:42.934095 28018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:54:42 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:42.950157 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.935071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44668)
0807 06:54:42.935534 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:42.935553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:42.935660 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:42.936024 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:42.936027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:42.936416 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:42.936611 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.936620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.937914 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.937920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.939060 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:42.939073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.940064 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.940070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:42.940313 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.941065 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:42.941082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:42.943401 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:42.947318 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.947323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.947326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.947871 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.948415 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:42.948419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:42.948421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:42.948550 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:42.949284 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:42.949287 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:42.949409 (+ 122us) client_negotiation.cc:769] Sending connection context
0807 06:54:42.949531 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 06:54:42.949763 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260807 06:54:42.950412 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.934981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.27.54.124:33059)
0807 06:54:42.935160 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.935165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.935619 (+ 454us) server_negotiation.cc:407] Connection header received
0807 06:54:42.935789 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.935794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.935872 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.935987 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.936755 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.937782 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.939200 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.939888 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.940000 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.943601 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.943607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.943612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.943648 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.947163 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.948033 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.948038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.948075 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.948274 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.948685 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.948691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.948695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.949151 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.949324 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.949687 (+ 363us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.949990 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:54:42.953367 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:42.935348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.27.54.125:40453)
0807 06:54:42.935511 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:42.935516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:42.935533 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:54:42.935587 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:42.935594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:42.935666 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:42.935790 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:42.937425 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.938315 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.939903 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:42.940602 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:42.940712 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:42.945698 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:42.945706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:42.945712 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:42.945756 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:42.949797 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.950966 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.950973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.950977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.951196 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:42.951674 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:42.951681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:42.951688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:42.952097 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:42.952436 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:42.952688 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 06:54:42.952978 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:54:42.957266 28018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: Bootstrap starting.
I20260807 06:54:42.959070 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:42.961316 28018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:42.962555 28018 log.cc:824] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:42.961869 27746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "49a591bc2a4a4da28949d7443a900392"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52952
I20260807 06:54:42.965649 28018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: No bootstrap required, opened a new log
I20260807 06:54:42.973012 28018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:42.973399 28018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468a6a7a3fa346c08210dbc49d97f7d3, State: Initialized, Role: FOLLOWER
I20260807 06:54:42.973860 28018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:42.974831 28030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:42.975185 28030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:42.975986 28018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:54:42.985885 28047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:54:42.986037 28047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:54:42.983712 27922 master_runner.cc:186] Error getting master registration for 127.27.54.126:38551: OK, instance_id { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" instance_seqno: 1786085681719103 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:54:42.992283 27922 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:54:43.034323 27912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 5
Aug 07 06:54:43 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
I20260807 06:54:43.052668 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.035311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44678)
0807 06:54:43.035530 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:43.035554 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:43.035675 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:43.036139 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:43.036145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:43.036718 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:43.036943 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:43.036960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.038475 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.038481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:43.039976 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:43.039992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.041029 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.041034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:43.041295 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.042042 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:43.042066 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:43.045103 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:43.049621 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:43.049630 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:43.049633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:43.050262 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:43.050701 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:43.050706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:43.050709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:43.050889 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:43.051556 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:43.051561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:43.051781 (+ 220us) client_negotiation.cc:769] Sending connection context
0807 06:54:43.051936 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 06:54:43.052254 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 06:54:43.052728 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.035382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.27.54.124:33059)
0807 06:54:43.035638 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.035644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.035670 (+ 26us) server_negotiation.cc:407] Connection header received
0807 06:54:43.035848 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.035854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.035942 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.036066 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.037115 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.038315 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.040140 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.040866 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.041014 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.045311 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:43.045319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:43.045325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:43.045370 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:43.049450 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.050399 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.050404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.050408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.050561 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.051006 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.051012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.051015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.051427 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:43.051600 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:43.052106 (+ 506us) server_negotiation.cc:299] Negotiation successful
0807 06:54:43.052356 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 06:54:43.057646 27912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: Bootstrap starting.
I20260807 06:54:43.061159 27912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:43.062270 27912 log.cc:824] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:43.065145 27912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: No bootstrap required, opened a new log
I20260807 06:54:43.072569 27912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:43.072974 27912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb5346326ef4411a0be9a0dcabc0de6, State: Initialized, Role: FOLLOWER
I20260807 06:54:43.073472 27912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:43.074517 28053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.074996 28053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:43.075776 27912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:54:43.087054 28068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:54:43.087337 28068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:54:43.084048 27799 master_runner.cc:186] Error getting master registration for 127.27.54.126:38551: OK, instance_id { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" instance_seqno: 1786085681719103 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:54:43.096179 27799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 06:54:43.109792 27957 tablet.cc:2369] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:54:43.141167 27791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } attempt: 5
Aug 07 06:54:43 dist-test-slave-3kk6 krb5kdc[27676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
I20260807 06:54:43.159958 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.142041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:33059 (local address 127.0.0.1:44686)
0807 06:54:43.142356 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:43.142377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:43.142520 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:43.143009 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:43.143016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:43.143563 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:43.143782 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:43.143796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.145341 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.145347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:43.146934 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:43.146951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.148034 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.148041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:43.148244 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.149027 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:43.149052 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:43.152065 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:43.156737 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:43.156744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:43.156749 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:43.157356 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:43.157850 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:43.157855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:43.157859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:43.158050 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:43.158842 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:43.158847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:43.159044 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 06:54:43.159195 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 06:54:43.159520 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260807 06:54:43.161007 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.142104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.27.54.124:33059)
0807 06:54:43.142310 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.142316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.142670 (+ 354us) server_negotiation.cc:407] Connection header received
0807 06:54:43.142749 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.142754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.142837 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.142941 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.143975 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.145203 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.147088 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.147858 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.148000 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.152285 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:43.152292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:43.152297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:43.152341 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:43.156534 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.157528 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.157534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.157538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.157707 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.158186 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.158191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.158195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.158580 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:43.158772 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:43.160326 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0807 06:54:43.160639 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 06:54:43.167469 27791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392: Bootstrap starting.
I20260807 06:54:43.171911 27791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:43.173295 27791 log.cc:824] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:43.177486 27791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392: No bootstrap required, opened a new log
I20260807 06:54:43.186122 27791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:43.186687 27791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a591bc2a4a4da28949d7443a900392, State: Initialized, Role: FOLLOWER
I20260807 06:54:43.187260 27791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:43.188719 28073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.189452 28073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:43.189929 27791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:43.190109 27746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 06:54:43.190330 27746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:43.193871 27746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:43.196769 27879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49a591bc2a4a4da28949d7443a900392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9bb5346326ef4411a0be9a0dcabc0de6"
I20260807 06:54:43.197307 27879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:43.201069 27879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a591bc2a4a4da28949d7443a900392 in term 1.
I20260807 06:54:43.202271 27714 leader_election.cc:302] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a591bc2a4a4da28949d7443a900392, 9bb5346326ef4411a0be9a0dcabc0de6; no voters:
I20260807 06:54:43.204013 27985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49a591bc2a4a4da28949d7443a900392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "468a6a7a3fa346c08210dbc49d97f7d3"
I20260807 06:54:43.204569 27985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [term 0 FOLLOWER]: Advancing to term 1
W20260807 06:54:43.208205 27719 tablet.cc:2369] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:54:43.209398 27985 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a591bc2a4a4da28949d7443a900392 in term 1.
I20260807 06:54:43.210166 27746 leader_election.cc:288] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [CANDIDATE]: Term 1 election: Requested vote from peers 9bb5346326ef4411a0be9a0dcabc0de6 (127.27.54.125:40453), 468a6a7a3fa346c08210dbc49d97f7d3 (127.27.54.124:33059)
I20260807 06:54:43.217970 28073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:43.238770 28073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [term 1 LEADER]: Becoming Leader. State: Replica: 49a591bc2a4a4da28949d7443a900392, State: Running, Role: LEADER
I20260807 06:54:43.239548 28073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } }
I20260807 06:54:43.257066 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a591bc2a4a4da28949d7443a900392. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.257687 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:43.271842 28090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:43.283141 28099 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 06:54:43.285727 27832 tablet.cc:2369] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:54:43.291262 28090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 06:54:43.372413 27985 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [term 1 FOLLOWER]: Refusing update from remote peer 49a591bc2a4a4da28949d7443a900392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:54:43.372922 27879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [term 1 FOLLOWER]: Refusing update from remote peer 49a591bc2a4a4da28949d7443a900392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:54:43.373678 28081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:54:43.374022 28073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:54:43.396210 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.281005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.27.54.124:33059)
0807 06:54:43.281262 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.281268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.346263 (+ 64995us) server_negotiation.cc:407] Connection header received
0807 06:54:43.346602 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.346611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.346765 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.346864 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.359949 (+ 13085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.361045 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.370310 (+ 9265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.371074 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.371184 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.385913 (+ 14729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:43.385920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:43.385925 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:43.385977 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:43.389989 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.392830 (+ 2841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.392840 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.392846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.393050 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.393754 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.393760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.393764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.394137 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:43.394378 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:43.395520 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0807 06:54:43.395814 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 06:54:43.397249 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.265830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.27.54.126:38551)
0807 06:54:43.266023 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.266030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.318626 (+ 52596us) server_negotiation.cc:407] Connection header received
0807 06:54:43.319109 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.319120 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.319237 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.319336 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.322636 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.324248 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.357710 (+ 33462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.358504 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.358682 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.382260 (+ 23578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:43.382271 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:43.382282 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:43.382334 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:43.386473 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.393000 (+ 6527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.393009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.393018 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.393194 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.393703 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.393709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.393713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.394141 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:43.394396 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:43.396590 (+ 2194us) server_negotiation.cc:299] Negotiation successful
0807 06:54:43.396889 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 06:54:43.399394 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:43.272335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.27.54.125:40453)
0807 06:54:43.272527 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.272534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.328833 (+ 56299us) server_negotiation.cc:407] Connection header received
0807 06:54:43.329140 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.329149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.329252 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.329348 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.351239 (+ 21891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.352546 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.365995 (+ 13449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:43.366629 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:43.366788 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:43.387629 (+ 20841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:43.387636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:43.387641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:43.387702 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:43.391893 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.394623 (+ 2730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.394630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.394633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.394847 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:43.395673 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:43.395678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:43.395681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:43.396113 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:43.396368 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:43.398727 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0807 06:54:43.399015 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 06:54:43.402177 28053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a591bc2a4a4da28949d7443a900392. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.402535 28053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:43.404121 28030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a591bc2a4a4da28949d7443a900392. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.404474 28030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:43.414379 28102 mvcc.cc:205] Tried to move back new op lower bound from 7315806959027068928 to 7315806958556164096. Current Snapshot: MvccSnapshot[applied={T|T < 7315806959027068928}]
I20260807 06:54:43.431456 28053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.431946 28053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:43.434456 28030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.434739 28030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:43.439364 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.439688 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:43.439978 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49a591bc2a4a4da28949d7443a900392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a591bc2a4a4da28949d7443a900392" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 38551 } } peers { permanent_uuid: "9bb5346326ef4411a0be9a0dcabc0de6" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 40453 } } peers { permanent_uuid: "468a6a7a3fa346c08210dbc49d97f7d3" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 33059 } } }
I20260807 06:54:43.440246 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392 [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:43.442118 28090 catalog_manager.cc:1347] Generated new cluster ID: 18bef24319d540deb7dcfb07f36177ed
I20260807 06:54:43.442234 28090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:43.471947 28090 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:43.474112 28090 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:43.504964 28090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49a591bc2a4a4da28949d7443a900392: Generated new TSK 0
I20260807 06:54:43.506314 28090 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:54:43.665861 28026 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.654951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.27.54.126:38551)
0807 06:54:43.655179 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.655186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.663661 (+ 8475us) server_negotiation.cc:407] Connection header received
0807 06:54:43.663925 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.663934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.664039 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.664164 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.665536 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52976: BlockingRecv error: recv got EOF from 127.0.0.1:52976 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 06:54:43.666886 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.661092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.27.54.125:40453)
0807 06:54:43.661354 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.661362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.664407 (+ 3045us) server_negotiation.cc:407] Connection header received
0807 06:54:43.664729 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.664738 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.664873 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.664966 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.666549 (+ 1583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39036: BlockingRecv error: recv got EOF from 127.0.0.1:39036 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 06:54:43.669551 28021 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.663964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.27.54.124:33059)
0807 06:54:43.664160 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.664167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.667494 (+ 3327us) server_negotiation.cc:407] Connection header received
0807 06:54:43.667785 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.667791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.667884 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.667983 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.669230 (+ 1247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44704: BlockingRecv error: recv got EOF from 127.0.0.1:44704 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 06:54:43.678453 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.676448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.27.54.125:40453)
0807 06:54:43.676638 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.676646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.676725 (+ 79us) server_negotiation.cc:407] Connection header received
0807 06:54:43.677009 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.677014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.677124 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.677249 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.678143 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39052: BlockingRecv error: recv got EOF from 127.0.0.1:39052 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 06:54:43.691740 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.689230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.27.54.125:40453)
0807 06:54:43.689494 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.689503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.690073 (+ 570us) server_negotiation.cc:407] Connection header received
0807 06:54:43.690319 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.690325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.690463 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.690584 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.691474 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39058: BlockingRecv error: recv got EOF from 127.0.0.1:39058 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260807 06:54:43.700811 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.697176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.27.54.125:40453)
0807 06:54:43.697431 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.697441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.699281 (+ 1840us) server_negotiation.cc:407] Connection header received
0807 06:54:43.699524 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.699529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.699618 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.699731 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.700530 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39074: BlockingRecv error: recv got EOF from 127.0.0.1:39074 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 06:54:43.714540 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.712103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.27.54.125:40453)
0807 06:54:43.712419 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.712428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.712990 (+ 562us) server_negotiation.cc:407] Connection header received
0807 06:54:43.713218 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.713223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.713368 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.713483 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.714297 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39078: BlockingRecv error: recv got EOF from 127.0.0.1:39078 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
W20260807 06:54:43.738754 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.736209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.27.54.125:40453)
0807 06:54:43.736451 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.736458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.737005 (+ 547us) server_negotiation.cc:407] Connection header received
0807 06:54:43.737253 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.737259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.737379 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.737506 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.738480 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39094: BlockingRecv error: recv got EOF from 127.0.0.1:39094 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260807 06:54:43.760787 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.758538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.27.54.125:40453)
0807 06:54:43.758799 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.758806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.759257 (+ 451us) server_negotiation.cc:407] Connection header received
0807 06:54:43.759437 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.759444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.759558 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.759696 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.760513 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39102: BlockingRecv error: recv got EOF from 127.0.0.1:39102 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 06:54:43.798739 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.796516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.27.54.125:40453)
0807 06:54:43.796737 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.796743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.797126 (+ 383us) server_negotiation.cc:407] Connection header received
0807 06:54:43.797311 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.797318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.797438 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.797582 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.798421 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39110: BlockingRecv error: recv got EOF from 127.0.0.1:39110 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260807 06:54:43.868316 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:43.866388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.27.54.125:40453)
0807 06:54:43.866586 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:43.866590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:43.867063 (+ 473us) server_negotiation.cc:407] Connection header received
0807 06:54:43.867270 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:43.867275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:43.867353 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:43.867454 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:43.868144 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39126: BlockingRecv error: recv got EOF from 127.0.0.1:39126 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 06:54:43.988178 28047 catalog_manager.cc:1259] Loaded cluster ID: 18bef24319d540deb7dcfb07f36177ed
I20260807 06:54:43.988294 28047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: loading cluster ID for follower catalog manager: success
I20260807 06:54:43.990993 28047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: acquiring CA information for follower catalog manager: success
I20260807 06:54:43.993017 28047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 468a6a7a3fa346c08210dbc49d97f7d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 06:54:44.002420 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:44.000038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.27.54.125:40453)
0807 06:54:44.000216 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:44.000220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:44.000678 (+ 458us) server_negotiation.cc:407] Connection header received
0807 06:54:44.000882 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:44.000891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:44.001046 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:44.001225 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:44.002111 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39142: BlockingRecv error: recv got EOF from 127.0.0.1:39142 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:54:44.089506 28068 catalog_manager.cc:1259] Loaded cluster ID: 18bef24319d540deb7dcfb07f36177ed
I20260807 06:54:44.089625 28068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: loading cluster ID for follower catalog manager: success
I20260807 06:54:44.092360 28068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: acquiring CA information for follower catalog manager: success
I20260807 06:54:44.094220 28068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9bb5346326ef4411a0be9a0dcabc0de6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:54:44.270828 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:44.264857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.27.54.125:40453)
0807 06:54:44.265141 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:44.265149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:44.265473 (+ 324us) server_negotiation.cc:407] Connection header received
0807 06:54:44.265661 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:44.265667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:44.265835 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:44.265945 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:44.266420 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.267258 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.268602 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.268998 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.269113 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:44.269614 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:44.270151 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:44.270272 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:44.270381 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 06:54:44.270452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 06:54:44.284004 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:44.277424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.27.54.124:33059)
0807 06:54:44.278521 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:44.278525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:44.278542 (+ 17us) server_negotiation.cc:407] Connection header received
0807 06:54:44.278638 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:44.278642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:44.278857 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:44.279025 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:44.279392 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.280161 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.281684 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.282075 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.282170 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:44.282788 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:44.283313 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:44.283426 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:44.283558 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 06:54:44.283627 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":844,"threads_started":1}
I20260807 06:54:44.308457 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:44.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.27.54.125:40453)
0807 06:54:44.292513 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:44.292523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:44.301058 (+ 8535us) server_negotiation.cc:407] Connection header received
0807 06:54:44.301310 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:44.301316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:44.301418 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:44.301486 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:44.302442 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.303570 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.305886 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.306572 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.306766 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:44.307548 (+ 782us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:44.307724 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:44.307793 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:44.308036 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 06:54:44.308129 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260807 06:54:44.309237 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:44.291675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.27.54.126:38551)
0807 06:54:44.300473 (+ 8798us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:44.300481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:44.300514 (+ 33us) server_negotiation.cc:407] Connection header received
0807 06:54:44.300683 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:44.300690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:44.300947 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:44.301180 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:44.302101 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.303398 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.306073 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.306756 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.306873 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:44.307640 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:44.308112 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:44.308223 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:44.308810 (+ 587us) server_negotiation.cc:299] Negotiation successful
0807 06:54:44.308904 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8632,"thread_start_us":8503,"threads_started":1}
I20260807 06:54:44.310132 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:44.292262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.27.54.124:33059)
0807 06:54:44.292685 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:44.292693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:44.302779 (+ 10086us) server_negotiation.cc:407] Connection header received
0807 06:54:44.303075 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:44.303081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:44.303185 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:44.303251 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:44.304219 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.305260 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.307533 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:44.308193 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:44.308293 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:44.309353 (+ 1060us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:44.309475 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:44.309551 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:44.309782 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 06:54:44.309866 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260807 06:54:44.320091 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27693
I20260807 06:54:44.321426 27781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:44.435638 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27693
W20260807 06:54:44.466637 27829 connection.cc:535] server connection from 127.0.0.1:39022 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:54:44.467449 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27799
I20260807 06:54:44.468634 27906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:44.581236 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27799
I20260807 06:54:44.613230 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 27922
I20260807 06:54:44.614562 28013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:44.725431 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 27922
2026-08-07T06:54:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5241 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 06:54:44 dist-test-slave-3kk6 krb5kdc[28225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 06:54:44 dist-test-slave-3kk6 krb5kdc[28225](info): set up 2 sockets
Aug 07 06:54:44 dist-test-slave-3kk6 krb5kdc[28225](info): commencing operation
krb5kdc: starting...
W20260807 06:54:46.080989 27865 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/test-user.keytab.
Aug 07 06:54:46 dist-test-slave-3kk6 krb5kdc[28225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T06:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T06:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.54.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
Entry for principal kudu/127.27.54.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab.
I20260807 06:54:46.469835 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:36029
--webserver_interface=127.27.54.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:36029,127.27.54.125:38101,127.27.54.124:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 06:54:46.845345 28245 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:54:46.845741 28245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:46.845824 28245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:46.845911 28245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:46.853662 28245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:46.853793 28245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:46.853857 28245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:46.853904 28245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:46.853961 28245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:46.864753 28245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:36029,127.27.54.125:38101,127.27.54.124:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.126.keytab
--principal=kudu/127.27.54.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.126:36029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.54.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:46.866231 28245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:46.867736 28245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:46.880877 28253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:46.880842 28254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:46.880791 28257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:46.881739 28245 server_base.cc:1034] running on GCE node
I20260807 06:54:46.882606 28245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:46.884245 28245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:46.885547 28245 hybrid_clock.cc:648] HybridClock initialized: now 1786085686885477 us; error 40 us; skew 500 ppm
I20260807 06:54:46.885973 28245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:46 dist-test-slave-3kk6 krb5kdc[28225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:46.889837 28245 init.cc:377] Logged in from keytab as kudu/127.27.54.126@KRBTEST.COM (short username kudu)
I20260807 06:54:46.891682 28245 webserver.cc:460] Webserver started at http://127.27.54.126:40569/ using document root <none> and password file <none>
I20260807 06:54:46.892517 28245 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:46.892675 28245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:46.893057 28245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:46.896589 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data/instance:
uuid: "8d1b789791a94c2794d397eaf7af1a7a"
format_stamp: "Formatted at 2026-08-07 06:54:46 on dist-test-slave-3kk6"
server_key: "c9c8f89f3d5736be1d869eb8dc8a4c26"
server_key_iv: "30574554315aa27716d9cb01de38ca14"
server_key_version: "encryptionkey@0"
I20260807 06:54:46.897552 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal/instance:
uuid: "8d1b789791a94c2794d397eaf7af1a7a"
format_stamp: "Formatted at 2026-08-07 06:54:46 on dist-test-slave-3kk6"
server_key: "c9c8f89f3d5736be1d869eb8dc8a4c26"
server_key_iv: "30574554315aa27716d9cb01de38ca14"
server_key_version: "encryptionkey@0"
I20260807 06:54:46.903677 28245 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 06:54:46.907639 28265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:46.909200 28245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 06:54:46.909435 28245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal
uuid: "8d1b789791a94c2794d397eaf7af1a7a"
format_stamp: "Formatted at 2026-08-07 06:54:46 on dist-test-slave-3kk6"
server_key: "c9c8f89f3d5736be1d869eb8dc8a4c26"
server_key_iv: "30574554315aa27716d9cb01de38ca14"
server_key_version: "encryptionkey@0"
I20260807 06:54:46.909698 28245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:46.939100 28245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:46.944279 28245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:46.944643 28245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:46.965828 28245 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.126:36029
I20260807 06:54:46.965848 28340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.126:36029 every 8 connection(s)
I20260807 06:54:46.967480 28245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/data/info.pb
I20260807 06:54:46.969129 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28245
I20260807 06:54:46.969362 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-0/wal/instance
I20260807 06:54:46.970022 27865 external_mini_cluster.cc:1428] Setting key e3e2d2b5177d1c9437acb492f6a0660c
I20260807 06:54:46.973346 28341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:46.985677 28341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:47 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.54.125@KRBTEST.COM; defaulting to no policy
I20260807 06:54:47.016072 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:46.990400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.27.54.126:36029)
0807 06:54:46.992339 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:46.992353 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:46.992414 (+ 61us) server_negotiation.cc:407] Connection header received
0807 06:54:46.995212 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:46.995218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:46.995535 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:46.995770 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:46.997689 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:46.999225 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.001918 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.002558 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.002813 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.007630 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:47.007640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:47.007654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:47.007690 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:47.010966 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.011827 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.011843 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.011845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.012012 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.012484 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.012492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.012495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.013082 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:47.013331 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:47.013939 (+ 608us) server_negotiation.cc:299] Negotiation successful
0807 06:54:47.014492 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":1690,"spinlock_wait_cycles":142592,"thread_start_us":1468,"threads_started":1}
I20260807 06:54:47.016045 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:46.990400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36029 (local address 127.0.0.1:42604)
0807 06:54:46.991427 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:46.991494 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:46.994631 (+ 3137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:46.995889 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:46.995900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:46.996843 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:46.997386 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:46.997415 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:46.999425 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:46.999433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.001697 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.001717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.002900 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.002912 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.003152 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.004004 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:47.004041 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:47.007316 (+ 3275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:47.011145 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.011160 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.011174 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.011664 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.012135 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.012141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.012145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.012362 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.013270 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:47.013279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:47.013494 (+ 215us) client_negotiation.cc:769] Sending connection context
0807 06:54:47.013824 (+ 330us) client_negotiation.cc:240] Negotiation successful
0807 06:54:47.014382 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":493,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.125@KRBTEST.COM" created.
I20260807 06:54:47.025954 28341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:54:47.027352 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.026858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40466)
0807 06:54:47.027060 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.027205 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260807 06:54:47.027652 28267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
I20260807 06:54:47.029497 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.029080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40474)
0807 06:54:47.029248 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.029352 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 06:54:47.029699 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38101: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab.
Entry for principal kudu/127.27.54.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab.
I20260807 06:54:47.065989 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:38101
--webserver_interface=127.27.54.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab
--principal=kudu/127.27.54.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:36029,127.27.54.125:38101,127.27.54.124:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:54:47.079077 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } attempt: 1
I20260807 06:54:47.080538 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.080021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40478)
0807 06:54:47.080245 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.080386 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 06:54:47.081807 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.081367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40482)
0807 06:54:47.081537 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.081625 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 06:54:47.082134 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38101: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
I20260807 06:54:47.150484 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } attempt: 2
I20260807 06:54:47.152137 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.151406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40490)
0807 06:54:47.151687 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.151939 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260807 06:54:47.153534 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.153007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40494)
0807 06:54:47.153204 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.153323 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 06:54:47.153860 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38101: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
I20260807 06:54:47.245217 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } attempt: 3
I20260807 06:54:47.246876 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.246172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40502)
0807 06:54:47.246451 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.246616 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260807 06:54:47.248131 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.247675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40516)
0807 06:54:47.247843 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.247938 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 06:54:47.248392 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38101: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
I20260807 06:54:47.391718 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } attempt: 4
I20260807 06:54:47.393301 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.392517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40518)
0807 06:54:47.392927 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.393081 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260807 06:54:47.394783 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.394278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40520)
0807 06:54:47.394471 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.394573 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 06:54:47.395095 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.125:38101: Network error: Client connection negotiation failed: client connection to 127.27.54.125:38101: connect: Connection refused (error 111)
W20260807 06:54:47.450511 28355 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:54:47.451000 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:47.451138 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:47.451280 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:47.459219 28355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:47.459342 28355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:47.459447 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:47.459533 28355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:47.459638 28355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:47.470785 28355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:36029,127.27.54.125:38101,127.27.54.124:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.125.keytab
--principal=kudu/127.27.54.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.125:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.54.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:47.472436 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:47.473985 28355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:47.487567 28361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:47.487560 28363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:47.487690 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:47.489012 28355 server_base.cc:1034] running on GCE node
I20260807 06:54:47.489917 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:47.491693 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:47.492950 28355 hybrid_clock.cc:648] HybridClock initialized: now 1786085687492855 us; error 69 us; skew 500 ppm
I20260807 06:54:47.493297 28355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:47 dist-test-slave-3kk6 krb5kdc[28225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:47.501415 28355 init.cc:377] Logged in from keytab as kudu/127.27.54.125@KRBTEST.COM (short username kudu)
I20260807 06:54:47.503461 28355 webserver.cc:460] Webserver started at http://127.27.54.125:39673/ using document root <none> and password file <none>
I20260807 06:54:47.504333 28355 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:47.504496 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:47.504889 28355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:47.508530 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data/instance:
uuid: "b640149d0c8647669befcd4bc903ca1b"
format_stamp: "Formatted at 2026-08-07 06:54:47 on dist-test-slave-3kk6"
server_key: "0fb526806e7174782c2a7848c3e0841f"
server_key_iv: "c2f862d34cb2fc26fad2108b5456fe7d"
server_key_version: "encryptionkey@0"
I20260807 06:54:47.509552 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal/instance:
uuid: "b640149d0c8647669befcd4bc903ca1b"
format_stamp: "Formatted at 2026-08-07 06:54:47 on dist-test-slave-3kk6"
server_key: "0fb526806e7174782c2a7848c3e0841f"
server_key_iv: "c2f862d34cb2fc26fad2108b5456fe7d"
server_key_version: "encryptionkey@0"
I20260807 06:54:47.515931 28355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 06:54:47.520344 28371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:47.522068 28355 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 06:54:47.522342 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal
uuid: "b640149d0c8647669befcd4bc903ca1b"
format_stamp: "Formatted at 2026-08-07 06:54:47 on dist-test-slave-3kk6"
server_key: "0fb526806e7174782c2a7848c3e0841f"
server_key_iv: "c2f862d34cb2fc26fad2108b5456fe7d"
server_key_version: "encryptionkey@0"
I20260807 06:54:47.522642 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:47.558070 28355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:47.563998 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:47.564373 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:47.585389 28355 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.125:38101
I20260807 06:54:47.585409 28440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.125:38101 every 8 connection(s)
I20260807 06:54:47.587001 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/data/info.pb
I20260807 06:54:47.588630 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28355
I20260807 06:54:47.588835 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-1/wal/instance
I20260807 06:54:47.589432 27865 external_mini_cluster.cc:1428] Setting key 259f0caa445b5e5206005262e9caae35
I20260807 06:54:47.592283 28441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:47.603478 28441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:47 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.54.124@KRBTEST.COM; defaulting to no policy
I20260807 06:54:47.633510 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.607564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.27.54.126:36029)
0807 06:54:47.608256 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:47.608266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:47.610012 (+ 1746us) server_negotiation.cc:407] Connection header received
0807 06:54:47.613663 (+ 3651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:47.613668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:47.613786 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:47.613897 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:47.616715 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.617548 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.620604 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.621075 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.621205 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.626469 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:47.626475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:47.626479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:47.626523 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:47.629899 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.630856 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.630864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.630872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.631009 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.631501 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.631513 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.631515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.631843 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:47.632034 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:47.632829 (+ 795us) server_negotiation.cc:299] Negotiation successful
0807 06:54:47.633083 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":790,"threads_started":1}
I20260807 06:54:47.634603 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.608323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36029 (local address 127.0.0.1:42606)
0807 06:54:47.609778 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.609846 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:47.613203 (+ 3357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:47.614257 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:47.614273 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:47.615306 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:47.616385 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.616421 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.617772 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.617781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.620367 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.620392 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.621408 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.621425 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.621861 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.622864 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:47.622903 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:47.626104 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:47.630090 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.630100 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.630129 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.630636 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.631141 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.631146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.631150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.631364 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.632002 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:47.632011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:47.632419 (+ 408us) client_negotiation.cc:769] Sending connection context
0807 06:54:47.632718 (+ 299us) client_negotiation.cc:240] Negotiation successful
0807 06:54:47.633337 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"thread_start_us":845,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.54.124@KRBTEST.COM" created.
I20260807 06:54:47.636943 28441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:47 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:47.653385 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.638004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40536)
0807 06:54:47.638216 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.638244 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:47.638342 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:47.639456 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:47.639461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:47.639959 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:47.640098 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.640107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.641567 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.641575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.642440 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.642451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.643443 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.643450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.643626 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.644358 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:47.644375 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:47.646908 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:47.650498 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.650504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.650509 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.650995 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.651411 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.651419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.651423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.651580 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.652243 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:47.652248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:47.652405 (+ 157us) client_negotiation.cc:769] Sending connection context
0807 06:54:47.652561 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 06:54:47.652896 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 06:54:47.653477 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.638112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.27.54.125:38101)
0807 06:54:47.639012 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:47.639019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:47.639047 (+ 28us) server_negotiation.cc:407] Connection header received
0807 06:54:47.639099 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:47.639105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:47.639303 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:47.639471 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:47.640287 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.641426 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.642596 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.643247 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.643386 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.647147 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:47.647160 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:47.647183 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:47.647213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:47.650318 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.651150 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.651159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.651161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.651291 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.651700 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.651708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.651711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.652093 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:47.652348 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:47.652714 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 06:54:47.653023 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":741,"thread_start_us":437,"threads_started":1}
I20260807 06:54:47.656580 28441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:54:47.657855 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.657350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41612)
0807 06:54:47.657542 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.657684 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260807 06:54:47.658109 28376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.659909 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.659531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41614)
0807 06:54:47.659689 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.659792 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 06:54:47.660121 28441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.54.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab.
Entry for principal kudu/127.27.54.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab.
I20260807 06:54:47.682547 27865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
/tmp/dist-test-taskHOwvC2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:42287
--webserver_interface=127.27.54.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab
--principal=kudu/127.27.54.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.54.84:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.54.126:36029,127.27.54.125:38101,127.27.54.124:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 06:54:47.694444 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } attempt: 5
Aug 07 06:54:47 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:47.709491 28441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 1
I20260807 06:54:47.710494 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.695610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.27.54.125:38101)
0807 06:54:47.695826 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:47.695830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:47.695855 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:54:47.695991 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:47.695995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:47.696063 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:47.696160 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:47.697315 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.698045 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.699698 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.700484 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.700628 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.704440 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:47.704445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:47.704448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:47.704486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:47.707432 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.708239 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.708244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.708247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.708387 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:47.708848 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:47.708854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:47.708859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:47.709193 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:47.709373 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:47.709900 (+ 527us) server_negotiation.cc:299] Negotiation successful
0807 06:54:47.710182 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 06:54:47.710798 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.710171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41626)
0807 06:54:47.710439 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.710573 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 06:54:47.711264 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.695355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:40540)
0807 06:54:47.695625 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.695651 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:47.695841 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:47.696228 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:47.696235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:47.696836 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:47.697109 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.697125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.698196 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.698205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.699515 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:47.699530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:47.700660 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:47.700664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:47.700845 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:47.701617 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:47.701672 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:47.704192 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:47.707621 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.707627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.707640 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.708074 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.708520 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:47.708526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:47.708529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:47.708718 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:47.709336 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:47.709341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:47.709560 (+ 219us) client_negotiation.cc:769] Sending connection context
0807 06:54:47.709760 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 06:54:47.710583 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 06:54:47.712173 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.711600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41638)
0807 06:54:47.711897 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.712002 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260807 06:54:47.712438 28441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.712610 28341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } has no permanent_uuid. Determining permanent_uuid...
I20260807 06:54:47.713965 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.713481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41648)
0807 06:54:47.713700 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.713818 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260807 06:54:47.715019 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.714690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41664)
0807 06:54:47.714828 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.714922 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260807 06:54:47.715260 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.766520 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 1
I20260807 06:54:47.768095 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.767530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41666)
0807 06:54:47.767790 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.767913 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 06:54:47.769204 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.768843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41678)
0807 06:54:47.769003 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.769087 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 06:54:47.769438 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.780786 28441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 2
I20260807 06:54:47.782245 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.781714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41692)
0807 06:54:47.781946 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.782062 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 06:54:47.783555 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.783140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41702)
0807 06:54:47.783326 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.783420 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 06:54:47.783820 28441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.837707 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 2
I20260807 06:54:47.839165 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.838503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41716)
0807 06:54:47.838799 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.838945 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 06:54:47.840255 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.839874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41726)
0807 06:54:47.840067 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.840140 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 06:54:47.840482 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.875072 28441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 3
I20260807 06:54:47.876749 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.875980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41732)
0807 06:54:47.876288 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.876399 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260807 06:54:47.878082 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.877623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41748)
0807 06:54:47.877870 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.877957 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260807 06:54:47.878315 28441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:47.946769 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 3
I20260807 06:54:47.948333 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.947637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41756)
0807 06:54:47.947934 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.948070 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260807 06:54:47.949690 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:47.949180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41760)
0807 06:54:47.949443 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:47.949546 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260807 06:54:47.949971 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:48.021529 28441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 4
I20260807 06:54:48.023020 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.022405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41768)
0807 06:54:48.022633 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.022848 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 06:54:48.024089 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.023744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:41782)
0807 06:54:48.023892 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.023975 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 06:54:48.024298 28441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
W20260807 06:54:48.066883 28450 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 06:54:48.067338 28450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 06:54:48.067474 28450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 06:54:48.067616 28450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 06:54:48.075621 28450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 06:54:48.075783 28450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 06:54:48.075925 28450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 06:54:48.076015 28450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 06:54:48.076133 28450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 06:54:48.086906 28450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.54.84:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.54.126:36029,127.27.54.125:38101,127.27.54.124:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/krb5kdc/kudu_127.27.54.124.keytab
--principal=kudu/127.27.54.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.54.124:42287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.54.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 05:58:21 UTC on ccycloud.quasar-rzbaah.root.comops.site
build id 4571904
ASAN enabled
I20260807 06:54:48.088428 28450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 06:54:48.089912 28450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 06:54:48.106046 28457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:48.106137 28458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 06:54:48.106094 28460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 06:54:48.107589 28450 server_base.cc:1034] running on GCE node
I20260807 06:54:48.108628 28450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 06:54:48.110265 28450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 06:54:48.111557 28450 hybrid_clock.cc:648] HybridClock initialized: now 1786085688111414 us; error 120 us; skew 500 ppm
I20260807 06:54:48.111971 28450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 06:54:48.116036 28450 init.cc:377] Logged in from keytab as kudu/127.27.54.124@KRBTEST.COM (short username kudu)
I20260807 06:54:48.118075 28450 webserver.cc:460] Webserver started at http://127.27.54.124:44963/ using document root <none> and password file <none>
I20260807 06:54:48.119030 28450 fs_manager.cc:359] Metadata directory not provided
I20260807 06:54:48.119208 28450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 06:54:48.119596 28450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 06:54:48.123325 28450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data/instance:
uuid: "4289aaf2e0de48beaed0cef2e0a27945"
format_stamp: "Formatted at 2026-08-07 06:54:48 on dist-test-slave-3kk6"
server_key: "0c3d3d6a32d7abf2692f8d2b99cbd649"
server_key_iv: "beca50a2708be55414ed4e5b45e2dfde"
server_key_version: "encryptionkey@0"
I20260807 06:54:48.124575 28450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal/instance:
uuid: "4289aaf2e0de48beaed0cef2e0a27945"
format_stamp: "Formatted at 2026-08-07 06:54:48 on dist-test-slave-3kk6"
server_key: "0c3d3d6a32d7abf2692f8d2b99cbd649"
server_key_iv: "beca50a2708be55414ed4e5b45e2dfde"
server_key_version: "encryptionkey@0"
I20260807 06:54:48.127215 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 4
I20260807 06:54:48.128458 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.127948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:43592)
0807 06:54:48.128172 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.128284 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 06:54:48.129539 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.129160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:43594)
0807 06:54:48.129323 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.129403 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 06:54:48.129787 28341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.54.124:42287: Network error: Client connection negotiation failed: client connection to 127.27.54.124:42287: connect: Connection refused (error 111)
I20260807 06:54:48.131707 28450 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260807 06:54:48.136054 28470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 06:54:48.137619 28450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 06:54:48.137863 28450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal
uuid: "4289aaf2e0de48beaed0cef2e0a27945"
format_stamp: "Formatted at 2026-08-07 06:54:48 on dist-test-slave-3kk6"
server_key: "0c3d3d6a32d7abf2692f8d2b99cbd649"
server_key_iv: "beca50a2708be55414ed4e5b45e2dfde"
server_key_version: "encryptionkey@0"
I20260807 06:54:48.138152 28450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 06:54:48.163946 28450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 06:54:48.169224 28450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 06:54:48.169577 28450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 06:54:48.191602 28450 rpc_server.cc:307] RPC server started. Bound to: 127.27.54.124:42287
I20260807 06:54:48.191622 28539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.54.124:42287 every 8 connection(s)
I20260807 06:54:48.193260 28450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/data/info.pb
I20260807 06:54:48.197382 27865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu as pid 28450
I20260807 06:54:48.197618 27865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHOwvC2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786085287136147-27865-0/minicluster-data/master-2/wal/instance
I20260807 06:54:48.197762 28540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 06:54:48.198228 27865 external_mini_cluster.cc:1428] Setting key 2617174018fd81d84305a701b3e1fc63
I20260807 06:54:48.210376 28540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
I20260807 06:54:48.229137 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.200459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.27.54.124:42287)
0807 06:54:48.201511 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.201530 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.201609 (+ 79us) server_negotiation.cc:407] Connection header received
0807 06:54:48.204067 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.204108 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.204860 (+ 752us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.205504 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.207337 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.209648 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.211482 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.212440 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.212738 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.219163 (+ 6425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.219172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.219185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.219247 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.223362 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.224386 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.224397 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.224408 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.224662 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.225206 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.225209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.225212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.225825 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.226184 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.226853 (+ 669us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.227444 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":440,"threads_started":1}
I20260807 06:54:48.233024 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.213999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.126:36029 (local address 127.0.0.1:45400)
0807 06:54:48.215033 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.215122 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:48.215351 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:48.215821 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:48.215829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:48.216605 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:48.216906 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.216917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.218288 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.218294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.219938 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.219959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.220917 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.220924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.221126 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.221770 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:48.221791 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:48.224814 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:48.229598 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.229609 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.229615 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.230161 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.230763 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.230768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.230773 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.231015 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.231677 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:48.231685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:48.231917 (+ 232us) client_negotiation.cc:769] Sending connection context
0807 06:54:48.232164 (+ 247us) client_negotiation.cc:240] Negotiation successful
0807 06:54:48.232607 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":495,"threads_started":1}
I20260807 06:54:48.233089 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.214007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.27.54.126:36029)
0807 06:54:48.214708 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.214716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.215226 (+ 510us) server_negotiation.cc:407] Connection header received
0807 06:54:48.215518 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.215523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.215595 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.215739 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.217132 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.218080 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.220162 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.220611 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.220847 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.225073 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.225081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.225086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.225145 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.229410 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.230341 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.230350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.230357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.230561 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.231192 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.231196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.231199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.231514 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.231704 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.232331 (+ 627us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.232560 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":270,"threads_started":1}
I20260807 06:54:48.235457 28540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.126@KRBTEST.COM
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:48.251717 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.233196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.27.54.126:36029)
0807 06:54:48.233457 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.233463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.233565 (+ 102us) server_negotiation.cc:407] Connection header received
0807 06:54:48.233895 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.233901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.233990 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.234122 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.235794 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.236894 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.238328 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.238902 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.239059 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.244811 (+ 5752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.244818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.244824 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.244875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.248055 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.249056 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.249063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.249067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.249320 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.249749 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.249754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.249758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.250208 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.250554 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.250882 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.251080 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":671,"threads_started":1}
I20260807 06:54:48.252514 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.236293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.125:38101 (local address 127.0.0.1:41666)
0807 06:54:48.236510 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.236532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:48.236647 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:48.237338 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:48.237343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:48.237799 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:48.237962 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.237971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.239273 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.239281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.240283 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.240297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.241072 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.241078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.241302 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.242027 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:48.242047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:48.244888 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:48.249061 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.249067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.249071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.249787 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.250250 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.250259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.250266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.250475 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.251336 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:48.251342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:48.251569 (+ 227us) client_negotiation.cc:769] Sending connection context
0807 06:54:48.251793 (+ 224us) client_negotiation.cc:240] Negotiation successful
0807 06:54:48.252145 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 06:54:48.252769 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.236422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.27.54.125:38101)
0807 06:54:48.237008 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.237013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.237031 (+ 18us) server_negotiation.cc:407] Connection header received
0807 06:54:48.237086 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.237090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.237164 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.237309 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.238202 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.239104 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.240457 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.240896 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.241055 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.245122 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.245128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.245131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.245170 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.248901 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.249954 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.249958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.249960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.250110 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.250642 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.250675 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.250678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.251174 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.251357 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.251956 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.252209 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":350,"threads_started":1}
I20260807 06:54:48.254004 28540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.124@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.54.125@KRBTEST.COM
I20260807 06:54:48.272550 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.255119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:43604)
0807 06:54:48.255282 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.255304 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:48.255438 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:48.255774 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:48.255780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:48.256337 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:48.256541 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.256558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.258131 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.258138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.259813 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.259828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.260939 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.260946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.261223 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.262146 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:48.262175 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:48.265399 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:48.269307 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.269317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.269325 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.269935 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.270397 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.270404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.270409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.270596 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.271339 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:48.271342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:48.271488 (+ 146us) client_negotiation.cc:769] Sending connection context
0807 06:54:48.271725 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 06:54:48.272108 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 06:54:48.272593 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.255048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.27.54.124:42287)
0807 06:54:48.255257 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.255262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.255371 (+ 109us) server_negotiation.cc:407] Connection header received
0807 06:54:48.255549 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.255553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.255611 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.255695 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.256719 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.257949 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.259995 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.260724 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.260880 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.265625 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.265630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.265675 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.265723 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.269137 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.270097 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.270102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.270104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.270244 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.270768 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.270773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.270776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.271196 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.271365 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.271859 (+ 494us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.272349 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":69}
I20260807 06:54:48.274257 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.253972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.27.54.125:38101)
0807 06:54:48.254159 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.254166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.254313 (+ 147us) server_negotiation.cc:407] Connection header received
0807 06:54:48.254458 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.254462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.254529 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.254720 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.256659 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.257949 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.260011 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.260725 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.260880 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.267077 (+ 6197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.267083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.267087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.267126 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.270581 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.271603 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.271609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.271613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.271823 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.272332 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.272339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.272343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.272768 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.273065 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.273604 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.273826 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 06:54:48.279513 28540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: Bootstrap starting.
I20260807 06:54:48.279794 27865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 06:54:48.283638 28540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:48.284929 28540 log.cc:824] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:48.282379 28302 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8d1b789791a94c2794d397eaf7af1a7a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45404
I20260807 06:54:48.288373 28540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: No bootstrap required, opened a new log
I20260807 06:54:48.297055 28540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.297480 28540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4289aaf2e0de48beaed0cef2e0a27945, State: Initialized, Role: FOLLOWER
I20260807 06:54:48.298061 28540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.299351 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.299823 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:48.300590 28540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:54:48.311376 28570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:54:48.311551 28570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:54:48.309005 28450 master_runner.cc:186] Error getting master registration for 127.27.54.126:36029: OK, instance_id { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" instance_seqno: 1786085686959947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:54:48.318002 28450 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:54:48.323527 28441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 5
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.125@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
I20260807 06:54:48.341521 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.324477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:43612)
0807 06:54:48.324730 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.324757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:48.324915 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:48.325479 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:48.325487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:48.326095 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:48.326331 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.326345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.328115 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.328120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.329560 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.329576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.330805 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.330812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.331098 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.331909 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:48.331928 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:48.334727 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:48.338673 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.338681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.338686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.339162 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.339620 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.339625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.339629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.339803 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.340423 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:48.340429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:48.340611 (+ 182us) client_negotiation.cc:769] Sending connection context
0807 06:54:48.340773 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 06:54:48.341119 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 06:54:48.341527 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.324580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.27.54.124:42287)
0807 06:54:48.324831 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.324837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.324862 (+ 25us) server_negotiation.cc:407] Connection header received
0807 06:54:48.325144 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.325152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.325274 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.325407 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.326535 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.327949 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.329747 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.330596 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.330839 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.335008 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.335015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.335020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.335076 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.338448 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.339320 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.339327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.339331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.339480 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.339931 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.339937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.339941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.340286 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.340489 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.340879 (+ 390us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.341115 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 06:54:48.348405 28441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: Bootstrap starting.
I20260807 06:54:48.353175 28441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:48.354732 28441 log.cc:824] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:48.358423 28441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: No bootstrap required, opened a new log
I20260807 06:54:48.370105 28441 raft_consensus.cc:348] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.370697 28441 raft_consensus.cc:729] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b640149d0c8647669befcd4bc903ca1b, State: Initialized, Role: FOLLOWER
I20260807 06:54:48.371335 28441 consensus_queue.cc:260] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.372889 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.373718 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:48.374235 28441 sys_catalog.cc:564] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: configured and running, proceeding with master startup.
W20260807 06:54:48.386399 28589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 06:54:48.386703 28589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 06:54:48.383423 28355 master_runner.cc:186] Error getting master registration for 127.27.54.126:36029: OK, instance_id { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" instance_seqno: 1786085686959947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 06:54:48.393471 28355 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 06:54:48.407020 28341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } attempt: 5
Aug 07 06:54:48 dist-test-slave-3kk6 krb5kdc[28225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786085686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.54.126@KRBTEST.COM for kudu/127.27.54.124@KRBTEST.COM
W20260807 06:54:48.422219 28479 tablet.cc:2369] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:54:48.426256 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.408261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.27.54.124:42287)
0807 06:54:48.408463 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.408469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.408575 (+ 106us) server_negotiation.cc:407] Connection header received
0807 06:54:48.408794 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.408800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.408881 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.409009 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.410054 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.410994 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.412312 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.413116 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.413306 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.417898 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.417906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.417910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.417963 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.421847 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.424014 (+ 2167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.424020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.424024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.424204 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.424707 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.424712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.424715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.425012 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.425188 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.425633 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.425873 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 06:54:48.426270 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.407911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.54.124:42287 (local address 127.0.0.1:43628)
0807 06:54:48.408464 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 06:54:48.408487 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 06:54:48.408646 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 06:54:48.409053 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 06:54:48.409058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 06:54:48.409662 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 06:54:48.409884 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.409899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.411131 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.411137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.412134 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 06:54:48.412148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.413332 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.413338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 06:54:48.413576 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.414327 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 06:54:48.414352 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 06:54:48.417581 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 06:54:48.423232 (+ 5651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.423242 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.423246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.423868 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.424351 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 06:54:48.424356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 06:54:48.424362 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 06:54:48.424586 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 06:54:48.425150 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 06:54:48.425155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 06:54:48.425340 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 06:54:48.425498 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 06:54:48.425829 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429}
I20260807 06:54:48.433034 28341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a: Bootstrap starting.
I20260807 06:54:48.437326 28341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a: Neither blocks nor log segments found. Creating new log.
I20260807 06:54:48.438905 28341 log.cc:824] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a: Log is configured to *not* fsync() on all Append() calls
I20260807 06:54:48.442932 28341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a: No bootstrap required, opened a new log
I20260807 06:54:48.451146 28341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.451689 28341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1b789791a94c2794d397eaf7af1a7a, State: Initialized, Role: FOLLOWER
I20260807 06:54:48.452260 28341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.453635 28594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.454394 28594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:48.455065 28341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 06:54:48.455165 28302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 06:54:48.455385 28302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:48.459810 28302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.463352 28405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1b789791a94c2794d397eaf7af1a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b640149d0c8647669befcd4bc903ca1b"
I20260807 06:54:48.464071 28405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:48.469170 28405 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1b789791a94c2794d397eaf7af1a7a in term 1.
I20260807 06:54:48.470340 28267 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d1b789791a94c2794d397eaf7af1a7a, b640149d0c8647669befcd4bc903ca1b; no voters:
I20260807 06:54:48.472775 28509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d1b789791a94c2794d397eaf7af1a7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4289aaf2e0de48beaed0cef2e0a27945"
I20260807 06:54:48.473493 28509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [term 0 FOLLOWER]: Advancing to term 1
I20260807 06:54:48.478932 28509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d1b789791a94c2794d397eaf7af1a7a in term 1.
I20260807 06:54:48.480129 28302 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [CANDIDATE]: Term 1 election: Requested vote from peers b640149d0c8647669befcd4bc903ca1b (127.27.54.125:38101), 4289aaf2e0de48beaed0cef2e0a27945 (127.27.54.124:42287)
I20260807 06:54:48.487615 28594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 06:54:48.495023 28594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [term 1 LEADER]: Becoming Leader. State: Replica: 8d1b789791a94c2794d397eaf7af1a7a, State: Running, Role: LEADER
I20260807 06:54:48.496546 28594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } }
I20260807 06:54:48.510521 28604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1b789791a94c2794d397eaf7af1a7a. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.511119 28604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:48.528800 28613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 06:54:48.531447 28621 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 06:54:48.554641 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.513169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.27.54.125:38101)
0807 06:54:48.513385 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.513392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.522447 (+ 9055us) server_negotiation.cc:407] Connection header received
0807 06:54:48.525507 (+ 3060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.525517 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.525703 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.525800 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.538284 (+ 12484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.539396 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.541658 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.542311 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.542423 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.546492 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.546500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.546504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.546558 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.550682 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.551610 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.551616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.551620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.551829 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.552354 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.552360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.552365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.552877 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.553189 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.553935 (+ 746us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.554227 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 06:54:48.554702 28613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260807 06:54:48.567457 28380 tablet.cc:2369] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 06:54:48.572846 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.514872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.27.54.124:42287)
0807 06:54:48.515136 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.515145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.524839 (+ 9694us) server_negotiation.cc:407] Connection header received
0807 06:54:48.527687 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.527698 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.527846 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.527974 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.536312 (+ 8338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.537450 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.544486 (+ 7036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.545315 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.545446 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.554904 (+ 9458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.554912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.554917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.554973 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.560107 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.569764 (+ 9657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.569772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.569778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.570027 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.570726 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.570735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.570741 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.571176 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.571465 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.572126 (+ 661us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.572445 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 06:54:48.576679 28509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [term 1 FOLLOWER]: Refusing update from remote peer 8d1b789791a94c2794d397eaf7af1a7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:54:48.577164 28405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [term 1 FOLLOWER]: Refusing update from remote peer 8d1b789791a94c2794d397eaf7af1a7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 06:54:48.577803 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:48.518109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.27.54.126:36029)
0807 06:54:48.518307 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.518316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.529761 (+ 11445us) server_negotiation.cc:407] Connection header received
0807 06:54:48.532498 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.532504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.532609 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.532689 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.541502 (+ 8813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.542638 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.550280 (+ 7642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:48.551170 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:48.551321 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:48.556960 (+ 5639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 06:54:48.556968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 06:54:48.556972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 06:54:48.557049 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0807 06:54:48.566926 (+ 9877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.573285 (+ 6359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.573291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.573294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.573499 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 06:54:48.574046 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 06:54:48.574051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 06:54:48.574054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 06:54:48.574497 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 06:54:48.574814 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:48.577122 (+ 2308us) server_negotiation.cc:299] Negotiation successful
0807 06:54:48.577406 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 06:54:48.577967 28594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:54:48.578779 28594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 06:54:48.623896 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1b789791a94c2794d397eaf7af1a7a. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.624311 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:48.632077 28624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.632633 28624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:48.633821 28613 catalog_manager.cc:1347] Generated new cluster ID: 35e8ebd34fa34368bbb7306ff23706a4
I20260807 06:54:48.633944 28613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 06:54:48.634574 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.634929 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 06:54:48.637440 28267 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.54.125:38101, user_credentials={real_user=slave}} blocked reactor thread for 59245.7us
I20260807 06:54:48.657302 28633 mvcc.cc:205] Tried to move back new op lower bound from 7315806980397604864 to 7315806980107653120. Current Snapshot: MvccSnapshot[applied={T|T < 7315806980397604864}]
I20260807 06:54:48.658967 28574 sys_catalog.cc:455] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1b789791a94c2794d397eaf7af1a7a. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.659338 28574 sys_catalog.cc:458] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:48.659530 28613 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 06:54:48.661439 28613 catalog_manager.cc:1504] Loading token signing keys...
I20260807 06:54:48.661517 28635 sys_catalog.cc:455] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.661935 28635 sys_catalog.cc:458] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 06:54:48.662883 28624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d1b789791a94c2794d397eaf7af1a7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1b789791a94c2794d397eaf7af1a7a" member_type: VOTER last_known_addr { host: "127.27.54.126" port: 36029 } } peers { permanent_uuid: "b640149d0c8647669befcd4bc903ca1b" member_type: VOTER last_known_addr { host: "127.27.54.125" port: 38101 } } peers { permanent_uuid: "4289aaf2e0de48beaed0cef2e0a27945" member_type: VOTER last_known_addr { host: "127.27.54.124" port: 42287 } } }
I20260807 06:54:48.663197 28624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a [sys.catalog]: This master's current role is: LEADER
I20260807 06:54:48.678103 28613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1b789791a94c2794d397eaf7af1a7a: Generated new TSK 0
I20260807 06:54:48.679466 28613 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 06:54:48.851423 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.828022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.27.54.125:38101)
0807 06:54:48.828332 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.828343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.847256 (+ 18913us) server_negotiation.cc:407] Connection header received
0807 06:54:48.849443 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.849451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.849584 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.849688 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.851137 (+ 1449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41698: BlockingRecv error: recv got EOF from 127.0.0.1:41698 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 06:54:48.854081 28544 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.834135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.27.54.124:42287)
0807 06:54:48.834363 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.834372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.849101 (+ 14729us) server_negotiation.cc:407] Connection header received
0807 06:54:48.852259 (+ 3158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.852268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.852384 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.852470 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.853806 (+ 1336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43646: BlockingRecv error: recv got EOF from 127.0.0.1:43646 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260807 06:54:48.857582 28546 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.846125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.27.54.126:36029)
0807 06:54:48.846420 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.846436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.855046 (+ 8610us) server_negotiation.cc:407] Connection header received
0807 06:54:48.855960 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.855969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.856072 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.856158 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.857292 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45416: BlockingRecv error: recv got EOF from 127.0.0.1:45416 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260807 06:54:48.923009 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.905473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.27.54.125:38101)
0807 06:54:48.905711 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.905719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.920368 (+ 14649us) server_negotiation.cc:407] Connection header received
0807 06:54:48.920684 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.920695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.920879 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.921020 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.922600 (+ 1580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41706: BlockingRecv error: recv got EOF from 127.0.0.1:41706 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 06:54:48.945755 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.938395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.27.54.125:38101)
0807 06:54:48.938738 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.938748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.943464 (+ 4716us) server_negotiation.cc:407] Connection header received
0807 06:54:48.943727 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.943737 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.943872 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.944024 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.945418 (+ 1394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41718: BlockingRecv error: recv got EOF from 127.0.0.1:41718 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260807 06:54:48.957506 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.954050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.27.54.125:38101)
0807 06:54:48.954321 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.954332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.955047 (+ 715us) server_negotiation.cc:407] Connection header received
0807 06:54:48.955462 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.955472 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.955610 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.955732 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.957098 (+ 1366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41732: BlockingRecv error: recv got EOF from 127.0.0.1:41732 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260807 06:54:48.985612 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.977531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.27.54.125:38101)
0807 06:54:48.978018 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.978029 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.983531 (+ 5502us) server_negotiation.cc:407] Connection header received
0807 06:54:48.983887 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:48.983894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:48.983996 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:48.984081 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:48.985345 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41740: BlockingRecv error: recv got EOF from 127.0.0.1:41740 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
W20260807 06:54:49.001752 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:48.998948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.27.54.125:38101)
0807 06:54:48.999255 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:48.999265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:48.999933 (+ 668us) server_negotiation.cc:407] Connection header received
0807 06:54:49.000213 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.000218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.000304 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.000450 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:49.001438 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41752: BlockingRecv error: recv got EOF from 127.0.0.1:41752 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260807 06:54:49.024386 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:49.021841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.27.54.125:38101)
0807 06:54:49.022111 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.022120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.022758 (+ 638us) server_negotiation.cc:407] Connection header received
0807 06:54:49.023018 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.023022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.023113 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.023247 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:49.024090 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41760: BlockingRecv error: recv got EOF from 127.0.0.1:41760 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 06:54:49.062984 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:49.060331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.27.54.125:38101)
0807 06:54:49.060595 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.060605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.061252 (+ 647us) server_negotiation.cc:407] Connection header received
0807 06:54:49.061526 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.061531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.061636 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.061788 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:49.062687 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41774: BlockingRecv error: recv got EOF from 127.0.0.1:41774 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260807 06:54:49.133431 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:49.130896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.27.54.125:38101)
0807 06:54:49.131139 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.131149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.131774 (+ 625us) server_negotiation.cc:407] Connection header received
0807 06:54:49.132037 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.132043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.132152 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.132287 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:49.133150 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41790: BlockingRecv error: recv got EOF from 127.0.0.1:41790 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 06:54:49.267460 28550 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 06:54:49.265384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.27.54.125:38101)
0807 06:54:49.265696 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.265701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.265975 (+ 274us) server_negotiation.cc:407] Connection header received
0807 06:54:49.266174 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.266183 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.266326 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.266440 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 06:54:49.267265 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41804: BlockingRecv error: recv got EOF from 127.0.0.1:41804 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
I20260807 06:54:49.313890 28570 catalog_manager.cc:1259] Loaded cluster ID: 35e8ebd34fa34368bbb7306ff23706a4
I20260807 06:54:49.314059 28570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: loading cluster ID for follower catalog manager: success
I20260807 06:54:49.316871 28570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: acquiring CA information for follower catalog manager: success
I20260807 06:54:49.318836 28570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4289aaf2e0de48beaed0cef2e0a27945: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:54:49.389036 28589 catalog_manager.cc:1259] Loaded cluster ID: 35e8ebd34fa34368bbb7306ff23706a4
I20260807 06:54:49.389151 28589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: loading cluster ID for follower catalog manager: success
I20260807 06:54:49.392004 28589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: acquiring CA information for follower catalog manager: success
I20260807 06:54:49.393918 28589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b640149d0c8647669befcd4bc903ca1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 06:54:49.534224 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:49.527766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.27.54.125:38101)
0807 06:54:49.528053 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.528059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.528342 (+ 283us) server_negotiation.cc:407] Connection header received
0807 06:54:49.528538 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.528542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.528731 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.528850 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:49.529399 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.530165 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.531648 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.532194 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.532340 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:49.532897 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:49.533471 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:49.533599 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:49.533799 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 06:54:49.533879 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 06:54:49.548584 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:49.541522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.27.54.124:42287)
0807 06:54:49.542176 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.542182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.542217 (+ 35us) server_negotiation.cc:407] Connection header received
0807 06:54:49.542446 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.542450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.542627 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.542878 (+ 251us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:49.543428 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.544214 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.545668 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.546099 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.546204 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:49.547003 (+ 799us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:49.547658 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:49.547854 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:49.548065 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 06:54:49.548200 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260807 06:54:49.567015 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:49.555024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.27.54.125:38101)
0807 06:54:49.555264 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.555274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.560783 (+ 5509us) server_negotiation.cc:407] Connection header received
0807 06:54:49.560994 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.560999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.561074 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.561182 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:49.561862 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.562986 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.564669 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.565373 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.565524 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:49.566154 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:49.566309 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:49.566417 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:49.566579 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 06:54:49.566705 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 06:54:49.567764 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:49.555464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.27.54.124:42287)
0807 06:54:49.560000 (+ 4536us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.560006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.560159 (+ 153us) server_negotiation.cc:407] Connection header received
0807 06:54:49.560586 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.560592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.560702 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.560806 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:49.561600 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.562730 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.564957 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.565403 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.565569 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:49.566808 (+ 1239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:49.566941 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:49.567016 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:49.567366 (+ 350us) server_negotiation.cc:299] Negotiation successful
0807 06:54:49.567452 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4365}
I20260807 06:54:49.568161 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 06:54:49.555024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.27.54.126:36029)
0807 06:54:49.559069 (+ 4045us) server_negotiation.cc:206] Beginning negotiation
0807 06:54:49.559078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 06:54:49.559764 (+ 686us) server_negotiation.cc:407] Connection header received
0807 06:54:49.560035 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 06:54:49.560043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 06:54:49.560316 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 06:54:49.560556 (+ 240us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 06:54:49.561389 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.562327 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.564876 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 06:54:49.565558 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 06:54:49.565706 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 06:54:49.566365 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 06:54:49.567008 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 06:54:49.567135 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 06:54:49.567574 (+ 439us) server_negotiation.cc:299] Negotiation successful
0807 06:54:49.567672 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3890,"thread_start_us":3767,"threads_started":1}
I20260807 06:54:49.574529 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28245
I20260807 06:54:49.575815 28334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:49.689495 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28245
W20260807 06:54:49.721661 28373 connection.cc:535] server connection from 127.0.0.1:40540 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 06:54:49.721635 28473 connection.cc:535] server connection from 127.0.0.1:43628 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 06:54:49.722543 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28355
I20260807 06:54:49.723807 28436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:49.839290 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28355
I20260807 06:54:49.867959 27865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu pid 28450
I20260807 06:54:49.869351 28534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 06:54:49.981024 27865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHOwvC2/build/asan/bin/kudu with pid 28450
2026-08-07T06:54:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5257 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10498 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (402767 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST